Recursive Language Models for Cross-Modal Inference: Integrating Visual and Textual Recursion

TL;DR: What if RLMs could recurse over both text and images, summarizing and decomposing not just words but visuals? By combining recursive prompt decomposition with visual in-context learning (VICL), we could enable multi-hop, multi-modal reasoning—imagine recursively answering questions about a comic book or illustrated manual.

Research Question: Can recursive inference strategies be extended to multi-modal prompts, allowing recursive decomposition and synthesis over both text and images for complex reasoning tasks?

Hypothesis: An RLM extended with visual in-context learning (as in Zhou et al., 2024) and recursive chunking of both textual and visual elements will outperform standard LLMs and LVLMs on tasks requiring joint text-image reasoning across long or complex inputs.

Experiment Plan: - Setup: Develop a multi-modal RLM architecture where recursive calls can process either text or images, leveraging intent-oriented summarization and demonstration composition (from VICL).

• Data: Use datasets with long-form illustrated content (e.g., instructional manuals, comics) and design new benchmarks for multi-hop, cross-modal reasoning.
• Measurements: Evaluate accuracy on complex QA tasks, efficiency, and the model’s ability to produce interpretable reasoning chains spanning modalities.
• Expected Outcomes: Enhanced performance on cross-modal, long-context tasks previously out of reach for standard RLMs or LVLMs.

References:

• Zhou, Y., Li, X., Wang, Q., & Shen, J. (2024). Visual In-Context Learning for Large Vision-Language Models. Annual Meeting of the Association for Computational Linguistics.