GapSketch: Visual-Anomaly Fusion for Missing Information Detection in Transformers

Building on AbsenceBench's revelation that transformers fail at detecting missing information due to their inability to attend to "gaps," GapSketch introduces a fundamentally different approach inspired by two key insights from the literature. First, it leverages the visual sketching paradigm from Hu et al.'s Visual Sketchpad work, where models draw auxiliary representations to facilitate reasoning. Second, it incorporates anomaly detection techniques from sources like Dekoninck et al.'s ConStat and Hu et al.'s SOWA framework, which excel at identifying deviations from expected patterns.

The core innovation is a three-stage process: (1) Gap Visualization - When processing a document, the model generates conceptual sketches of expected but missing elements (e.g., drawing placeholder boxes for omitted data points, using symbolic representations for missing logical connectors), inspired by Sketchpad's visual reasoning approach. (2) Anomaly-Aware Attention - These visual gap representations are encoded into the attention mechanism as synthetic keys, allowing transformers to "attend to absences" through their proxies. This draws from anomaly detection methods that identify statistical deviations (like those in Zhou et al.'s log parsing and Yang et al.'s network traffic monitoring). (3) Cross-Modal Validation - The framework uses the anomaly detection principles from sources like Kumar et al.'s vital sign monitoring to validate whether identified gaps represent true absences versus just unexpected but valid content.

This approach diverges from existing work by transforming the abstract problem of "attending to nothing" (as identified in AbsenceBench) into a concrete problem of attending to visual representations of absences. While Mousavian et al. used LLMs to detect subtle gender biases (missing fairness), and Biran et al. analyzed multi-hop failures (missing connections), GapSketch explicitly makes the missing information visible and attendable. The fusion of visual sketching with anomaly detection represents a novel synthesis across multiple domains - creating a new paradigm where gaps become first-class objects in the attention mechanism rather than invisible limitations.

The potential impact is significant: by enabling transformers to detect missing information, we could improve applications ranging from clinical decision support (addressing Hager et al.'s findings about LLMs missing critical patient data) to fraud detection (enhancing Otuburun's work by identifying omitted suspicious details). This research opens new avenues for developing "absence-aware" AI systems that can not only recall what's present but critically identify what's missing.

References:

1. Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement. Maryam Mousavian, Zahra Abbasiantaeb, Mohammad Aliannejadi, Fabio Crestani (2025). International Conference on the Theory of Information Retrieval.
2. Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke S. Zettlemoyer, Noah A. Smith, Ranjay Krishna (2024). Neural Information Processing Systems.
3. ConStat: Performance-Based Contamination Detection in Large Language Models. Jasper Dekoninck, Mark Niklas Müller, Martin T. Vechev (2024). Neural Information Processing Systems.
4. Evaluation and mitigation of the limitations of large language models in clinical decision-making. P. Hager, F. Jungmann, R. Holland, K. Bhagat, I. Hubrecht, M. Knauer, J. Vielhauer, M. Makowski, R. Braren, G. Kaissis, D. Rueckert (2024). Nature Network Boston.
5. Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries. Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, Amir Globerson (2024). Conference on Empirical Methods in Natural Language Processing.
6. An Adaptive Transformer-Autoencoder Framework for Reliable Anomaly Detection in Patient Vital Signs. Sunil Kumar V, Zahraa Alkhafajy, Abhishek Kumar Verma, S. S. Bhaviya, C. Sudhakar (2025). 2025 Third International Conference on Networks, Multimedia and Information Technology (NMITCON).
7. Leveraging Large Language Models and BERT for Log Parsing and Anomaly Detection. Yihan Zhou, Yan Chen, Xuanming Rao, Yukang Zhou, Yuxin Li, Chao Hu (2024). Mathematics.
8. Research on Cloud Platform Network Traffic Monitoring and Anomaly Detection System based on Large Language Models. Ze Yang, Yihong Jin, Juntian Liu, Xinhe Xu, Yihan Zhang, Shuyang Ji (2025). 2025 IEEE 7th International Conference on Communications, Information System and Computer Engineering (CISCE).
9. SOWA: Adapting Hierarchical Frozen Window Self-Attention to Visual-Language Models for Better Anomaly Detection. Zongxiang Hu, Zhaosheng Zhang (2024). arXiv.org.
10. Real-Time Fraud Detection Using Large Language Models: A Context-Aware System for Mitigating Social Engineering Threats. Irhimefe Otuburun (2025). World Journal of Advanced Research and