Symbolic Sequence Augmentation: Bridging Latent Tokens and Interpretable Visual Reasoning

TL;DR: Can we make the model spell out its thinking like step-by-step instructions? By translating latent visual tokens into symbolic sequences, we could boost interpretability and compositional robustness.

Research Question: Will augmenting latent visual reasoning tokens with self-supervised symbolic sequence extraction improve interpretability and compositional generalization in visual reasoning models?

Hypothesis: Mapping latent visual reasoning tokens to symbolic, human-interpretable sequences (as in Martinez Pozos & Meza, 2025) will enable models to better handle compositional and out-of-distribution challenges, and provide interpretable rationales for their decisions.

Experiment Plan: Extend the latent token framework to generate symbolic reasoning sequences using a self-supervised decoder transformer. Evaluate on tasks requiring explicit reasoning steps (e.g., visual math, chart reasoning, abstract pattern completion). Measure improvements in compositional generalization and human interpretability, using metrics and user studies. Analyze attention maps and symbolic outputs to probe which visual abstractions are being captured and how they translate into reasoning steps.

References:

• Martinez Pozos, V. S., & Meza, I. V. (2025). Extracting Symbolic Sequences from Visual Representations via Self-Supervised Learning. arXiv.org.
• He, W., Xi, Z., Zhao, W., Fan, X., Ding, Y., Shan, Z., Gui, T., Zhang, Q., & Huang, X. (2024). Distill Visual Chart Reasoning Ability from LLMs to MLLMs. Conference on Empirical Methods in Natural Language Processing.
• Li, K., Shang, C., Karlinsky, L., Feris, R., Darrell, T., & Herzig, R. (2025). Latent Implicit Visual Reasoning.