Critique Markets for Discovery: Turning Reviewer Feedback into a Resource Allocation Signal

TL;DR: Give your AI scientist a picky reviewer who pays “attention tokens” only for genuinely good ideas—then let the AI chase what the reviewer rewards. We’ll embed an automated reviewer and treat its scores as a budget signal to schedule how the agent spends cycles on reading, coding, or synthesis, probing the observed linear scaling of findings in Kosmos.

Research Question: Can integrating automated critique and market-like allocation improve the marginal value of each additional Kosmos cycle, and where does the linear scaling of valuable findings begin to saturate?

Hypothesis: Turning reviewer feedback into a market signal that controls exploration-exploitation and cycle allocation will (i) increase novelty-adjusted accuracy per cycle and (ii) reveal a domain-dependent scaling law and saturation point earlier than naïve linear extrapolation suggests.

Experiment Plan: - Setup:

• Base system: Kosmos-like multi-agent loop.
• Automated reviewer: AI Scientist’s reviewer calibrated on held-out human reviews and rubric-anchored scores (novelty, rigor, clarity, evidence).
• Critique market: each hypothesis/analysis “bids” for cycles; reviewer returns “payouts” that drive a bandit-style scheduler to allocate the next cycles to the most promising lines of inquiry.
• Data/Materials: Two domains with different evidence dynamics: (1) plant science (Aleks case; structured data, interpretable models), (2) materials (text-heavy with code experiments). Optionally include a qualitative synthesis task (ethics literature) to test generality.
• Measurements:
  • Slope of valuable findings vs. cycles with and without critique markets.
  • Reviewer-human agreement; stability of scores over time.
  • Saturation detection: cycles at which marginal payouts drop below threshold; compare across domains.
• Expected Outcomes:
  • Improved novelty-adjusted accuracy per unit time.
  • Empirical characterization of scaling (linear in early regime; sublinear plateau) and identification of intervention points (e.g., literature refresh or role rotation) that delay saturation.

References: ['Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024). The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv.org.', 'Mitchener, L., Yiu, A., Chang, B., Bourdenx, M., Nadolski, T., Sulovari, A., et al. (2025). Kosmos: An AI Scientist for Autonomous Discovery. Preprint.', 'Jin, D., Gunner, N., Carvajal Janke, N., Baruah, S., Gold, K., & Jiang, Y. (2025). Aleks: AI powered Multi Agent System for Autonomous Scientific Discovery via Data-Driven Approaches in Plant Science. arXiv.org.', 'Eger, S., Cao, Y., D’Souza, J., Geiger, A., Greisinger, C., Gross, S., Hou, Y., Krenn, B., Lauscher, A., Li, Y., Lin, C., Moosavi, N., Zhao, W., & Miller, T. (2025). Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation. arXiv.org.', 'Reddy, C. K., & Shojaee, P. (2024). Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges. AAAI Conference on Artificial Intelligence.']