Recursive Language Models Meet Tree-Organized Retrieval: RAPTOR-RLM Hybrid for Document-Scale Synthesis

TL;DR: What if RLMs could not just recursively split prompts, but also climb and retrieve from a summary tree, choosing to go broad or deep for each recursive call? By marrying RAPTOR’s tree-based retrieval with RLM’s decomposition, we could allow models to flexibly synthesize information at multiple levels of abstraction from massive corpora.

Research Question: Can integrating tree-organized, recursive retrieval (RAPTOR) with RLMs enable more effective and scalable document-level or multi-document synthesis?

Hypothesis: A hybrid model that recursively decomposes inputs and queries a summary tree at each step (as in RAPTOR, Sarthi et al., 2024) will outperform both vanilla RLMs and RAPTOR in tasks demanding both fine-grained and high-level synthesis across long or multiple documents.

Experiment Plan: - Setup: Implement an RLM variant where recursive calls may select nodes from a RAPTOR-style summary tree for retrieval and further recursion.

• Data: Use multi-document summarization and QA benchmarks, e.g., QuALITY, enterprise datasets (Godbole et al., 2024).
• Measurements: Compare holistic answer quality, retrieval efficiency, and ability to integrate information spanning multiple documents.
• Expected Outcomes: The RAPTOR-RLM hybrid should show significant gains in efficiency and accuracy for tasks requiring both deep dives and broad synthesis.

References:

• Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., & Manning, C. D. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. International Conference on Learning Representations.
• Godbole, A., George, J. G., & Shandilya, S. (2024). Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications. arXiv.org.