Coherence-as-a-Resource: A Control-Theoretic Scheduler for Long-Running AI Scientists

TL;DR: Think of an AI scientist’s “focus” like a battery that drains and needs smart recharging to keep working coherently for days, not hours. We’ll model coherence as a measurable state variable and use a controller to schedule when to read, code, summarize, or “checkpoint and compress” knowledge. Initial experiment: compare Kosmos-style runs with and without the scheduler over 12–24 hours and test whether we can extend depth (cycles) without increasing factual errors or goal drift.

Research Question: Can we formalize and control the “coherence budget” of an AI scientist to maintain goal alignment and reasoning quality over longer horizons than Kosmos, thereby increasing the depth of discoveries per run?

Hypothesis: A closed-loop “coherence controller” that monitors proxies of drift (e.g., topic divergence, reviewer score decay, code failure rates) and triggers targeted interventions (summarization, memory distillation, or role rebalancing) will preserve accuracy and reduce goal drift, enabling more valuable findings per additional cycle than an open-loop schedule.

Experiment Plan: - Setup: Implement a controller around a Kosmos-like architecture with (i) online coherence signals (semantic drift from the objective, claim-to-citation consistency, execution stability), (ii) interventions (world-model compression, targeted literature refresh, code refactoring sweeps, domain-expert subagent escalation), and (iii) a scheduler that allocates time across data analysis, reading, and synthesis.

• Data/Materials: Use two domains to stress generality: plant phenotyping (Aleks dataset/tasks) and materials text + crystallographic corpora (LLaMat-style corpora). Include one clinical review-like task to test safety constraints (e.g., summarizing NAFLD histopath evidence).
• Measurements:
  • Coherence metrics: topic drift (semantic distance to objective), claim-to-citation alignment, code success rate, and memory retrieval precision.
  • Scientific value: independent expert ratings (as in Kosmos), count of nontrivial insights, and novelty (literature distance).
  • Scaling: marginal valuable findings per extra cycle (testing whether Kosmos’s linear scaling extends or saturates).
• Expected Outcomes:
  • With control, long runs (≥20 cycles) maintain accuracy at or above Kosmos baselines and yield a higher slope of valuable findings vs. cycles.
  • Ablations (no controller; controller without distillation) show increased drift and lower reviewer scores over time.

References: ['Mitchener, L., Yiu, A., Chang, B., Bourdenx, M., Nadolski, T., Sulovari, A., et al. (2025). Kosmos: An AI Scientist for Autonomous Discovery. Preprint.', 'Jin, D., Gunner, N., Carvajal Janke, N., Baruah, S., Gold, K., & Jiang, Y. (2025). Aleks: AI powered Multi Agent System for Autonomous Scientific Discovery via Data-Driven Approaches in Plant Science. arXiv.org.', 'Reddy, C. K., & Shojaee, P. (2024). Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges. AAAI Conference on Artificial Intelligence.', 'Zheng, T., Deng, Z., Tsang, H. T., Wang, W., Bai, J., Wang, Z., Song, Y. (2025). From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery. arXiv.org.', 'Oksanen, J., Lucero, A., & Hämäläinen, P. (2025). LLMCode: Evaluating and Enhancing Researcher-AI Alignment in Qualitative Analysis. arXiv.org.']