Injecting meta intelligence about research in LLMs

TL;DR: LLMs can follow specific instructions, write code, and use tools given detailed usages. Existing LLM agents are highly capable, and a lot of research agents emerged, but none of them really produce high quality research. Can we inject knowledge about good research standards into LLMs?

Hypothesis: LLMs do not have the "meta intelligence" to do good research, for example:

• Knowing when the existing information and resources are limited and being able to search for external sources
• Have good practices like proper resource allocation and designing good ablations,
• Domain-specific standards, like write maintainable code in CS

Research plan:

1. Curate dataset of good standards in research. One possible approach is to create samples of (research scenario, action) pairs, where given an intermediate research snapshot, we provide the best action (or actions, since it's unclear that there is always one best action to take in research).
2. Train a research orchestrator using the created dataset. The orchestrator will be serving as a "research manager", and it will make decisions about what action to take for next steps.
3. Connect the research orchestrator with coding agents that will only focus on completing well-defined and concrete tasks, i.e., no open-endedness.

Note: Optimizing the prompts and context engineering for existing research agents can potentially lead to significant improvement, and is much easier to implement and experiment with compared to the orchestrator training approach. It is a plausible approach for AI/CS research, but the concern here is that it's unlikely one can exhaust the good standards in all domains and connect that effectively to LLM agents.

Challenges: Aside from the difficulty for curating a high quality dataset for representing good research standards, it's also unclear how we can define a good reward signal, e.g., how to evaluate a series of actions.