Small Model-Guided Adaptive Attention Matching for KV Cache Compaction

TL;DR: Let's have a small, cheap model help the big model decide which information is really important to keep when compressing the cache. The experiment would adapt the SmallKV technique—using a small model’s attention patterns to guide the selection and weighting of tokens during fast attention matching.

Research Question: Can leveraging a small model’s global attention patterns to guide attention-matching KV compaction improve the retention of marginal or contextually salient tokens?

Hypothesis: Using a small model’s attention matrices to inform compaction will improve the preservation of marginally important tokens and dynamically adapt to saliency shifts, outperforming single-model attention matching.

Experiment Plan: Implement a dual-model pipeline: for each sequence, first run a small, efficient LLM to compute attention saliency scores. Use these scores to weight or select tokens in the attention-matching compaction step for the larger LLM. Benchmark against single-model compaction on dynamic-context tasks (e.g., BBH, LongBench). Measure effects on marginal token retention, downstream accuracy, and throughput.

References:

• Zhao, Y., Peng, Y., Nguyen, C.-T., Li, Z., Wang, X., Zhao, H., & Fu, X. (2025). SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference. arXiv.org.
• Zweiger, A., Fu, X., Guo, H., & Kim, Y. (2026). Fast KV Compaction via Attention Matching.