Continual Test-Time RL with Memory-Augmented LLMs for Open Problems

TL;DR: What if the model could remember and reuse what it learned from previous problems, evolving its strategies over time? Let’s add an explicit memory module to ThetaEvolve for lifelong learning across problem sequences.

Research Question: Does equipping LLMs with a persistent, external memory during test-time RL enable more efficient continual adaptation and cumulative learning across a sequence of open optimization problems?

Hypothesis: A memory-augmented LLM will learn to recall and reuse successful strategies, leading to faster adaptation and better performance on new problems compared to stateless test-time RL.

Experiment Plan: Integrate an external memory (e.g., key-value store of past RL checkpoints, strategies, or reward histories) into the ThetaEvolve pipeline. Across a curriculum of open problems, allow the model to read from and write to memory at each adaptation step. Assess metrics such as time to best-known bound, number of RL steps needed, and cross-task transfer. Ablate memory to verify benefits.

References:

• 1. Wang, Y., et al. (2025). ThetaEvolve: Test-time Learning on Open Problems.
• 2. Iftee, M. A. R., Mahjabin, W., Ekka, A., & Das, S. (2024). MoE-TTA: Enhancing Continual Test-Time Adaptation for Vision-Language Models through Mixture of Experts. 2024 27th International Conference on Computer and Information Technology (ICCIT).