Diverse and Challenging Open-Problem Datasets for Program-Evolving LLMs

TL;DR: Let’s build a next-generation benchmark: a curated collection of diverse open problems, including program synthesis, combinatorial, and geometric tasks, to robustly evaluate evolving LLMs like ThetaEvolve.

Research Question: How does the performance and evolving capability of LLMs trained with test-time RL generalize on a broad, challenging suite of open problems beyond those seen in ThetaEvolve?

Hypothesis: A more diverse benchmark will expose the limitations and strengths of existing methods, and drive innovations in exploration, generalization, and adaptation strategies.

Experiment Plan: Assemble and release a new open-problem suite, combining real-world program synthesis, combinatorial optimization (e.g., graph coloring, scheduling), and geometric problems. Evaluate ThetaEvolve and other baselines on this suite, tracking solution quality, adaptation speed, and exploration diversity. Use the results to inform new RL reward structures or curriculum strategies.

References:

• 1. Wang, Y., et al. (2025). ThetaEvolve: Test-time Learning on Open Problems.
• 2. Shi, K., Hong, J., Zaheer, M., Yin, P., & Sutton, C. (2023). ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis. International Conference on Learning Representations.
• 3. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., & Sutton, C. (2021). Program Synthesis with Large Language Models. arXiv.org.