Visual Enrichment and Recall: Investigating Multiple Information Streams in Finetuned Vision Transformers

Inspired by Nief et al.'s 2025 discovery of dual pathways in finetuned language models—an 'enrichment' stream extracting relational information early and a 'recall' stream retrieving it before prediction—this research proposes to investigate if a similar mechanism exists in finetuned vision transformers (ViTs). While prior work like Ahmed and Jalal (2024) explored multi-branch ViT architectures focusing on input processing, this idea focuses on the internal information flow of learned knowledge within a single finetuned ViT. Using the 'dynamic weight-grafting' technique from Nief et al., the study would probe where and how specific visual knowledge (e.g., identifying hydatid cysts in CT scans as in Sağık and Gumus (2025)) is stored and processed across layers. The goal is to determine if early layers perform feature enrichment on relevant patches and if later layers have a distinct recall mechanism that consolidates this information for final classification. This challenges the prevailing assumption that ViTs process visual data in a monolithic, single-pass manner and could pioneer a new field of mechanistic interpretability in computer vision. Such insights could enable the design of more efficient ViTs with explicit enrichment and recall modules and improve transparency and trust in high-stakes applications like medical imaging by tracing diagnostic decisions to specific internal pathways.

References:

1. Multiple Streams of Relation Extraction: Enriching and Recalling in Transformers. Todd Nief, David Reber, Sean Richardson, Ari Holtzman (2025). arXiv.org.
2. RGB-D Scene Classification: A Unified Framework with Vision Transformers and Contextual Models. Muhammad Waqas Ahmed, Ahmad Jalal (2024). 2024 3rd International Conference on Emerging Trends in Electrical, Control, and Telecommunication Engineering (ETECTE).
3. Vision Transformers-Based Deep Feature Generation Framework for Hydatid Cyst Classification in Computed Tomography Images.. Metin Sağık, Abdurrahman Gumus (2025). Journal of imaging informatics in medicine.