Meta-Learning Recursive Language Models for One-Shot Program Synthesis

TL;DR: What if RLMs could learn to write better recursive decompositions after seeing just a handful of examples, like humans picking up new rules fast? We could train a meta-RLM that adapts its recursive strategies to new prompt structures or novel domains after a single demonstration, mimicking the few-shot generalization in human rule-learning.

Research Question: Can recursive language models be meta-trained to rapidly adapt their decomposition and recursive invocation strategies across new domains, thus enabling one-shot program synthesis and improved compositional generalization?

Hypothesis: Meta-learned RLMs will outperform standard RLMs and existing neural program synthesis methods (e.g., Nye et al., 2020) in few-shot settings, displaying faster adaptation and better generalization to unseen prompt structures or logic.

Experiment Plan: Construct a suite of synthetic and real-world tasks (e.g., SCAN, code translation, number word parsing) requiring recursive decomposition. Meta-train an RLM using a protocol akin to meta-learning, where in each episode it must learn a new decomposition strategy from a small set of demonstration prompts. Compare to baseline RLMs (no meta-learning) and neural program synthesis models (Nye et al., 2020). Measure adaptation speed, generalization to out-of-distribution prompt structures, and sample efficiency. Analyze qualitative differences in the learned recursive programs.

References:

• Nye, M., Solar-Lezama, A., Tenenbaum, J., & Lake, B. (2020). Learning Compositional Rules via Neural Program Synthesis. Neural Information Processing Systems.
• Recursive Language Models (2025). arXiv.org.
• Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., & Harchaoui, Z. (2024). From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models. Trans. Mach. Learn. Res.