SparseSight: Token-Sparse Encoder-Free VLMs with Image-Biased Decoding

Develop an encoder-free VLM (building on EVE; Diao et al., 2024) that learns to select a minimal set of informative visual tokens before decoding, inspired by VisionZip’s token redundancy findings (Yang et al., 2024). During generation, perform image-biased decoding (IBD; Zhu et al., 2024) by contrasting a conventional and an image-biased forward pass to amplify image-consistent tokens and suppress language-prior hallucinations. Reconciles two seemingly conflicting directions—removing vision encoders for flexibility (EVE) and shortening visual token streams for efficiency (VisionZip)—and adds a decoding-time fix (IBD) rather than relying solely on training. Optionally pretrain/fine-tune with high-quality synthetic instruction data (ALLaVA; Chen et al., 2024) to strengthen small, efficient variants. EVE shows the feasibility of encoder-free training; VisionZip shows longer visual sequences are often unnecessary; IBD shows decoding can reduce hallucinations without extra data. SparseSight integrates all three into a single system. Achieves faster prefilling and lower memory with encoder-free simplicity, while IBD curbs text-biased hallucination—two pain points for practical deployment. A path to compact, accurate VLMs for edge devices and interactive systems, particularly where latency and truthfulness are both critical.

References:

1. ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models. Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, Benyou Wang (2024).
2. VisionZip: Longer is Better but Not Necessary in Vision Language Models. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, Jiaya Jia (2024). Computer Vision and Pattern Recognition.
3. Unveiling Encoder-Free Vision-Language Models. Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, Xinlong Wang (2024). Neural Information Processing Systems.
4. IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding. Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, Jun Liu (2024). 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).