Hybrid Attention Matching with Query-Agnostic Context Reconstruction

TL;DR: Imagine combining the speed of attention-matching compaction with the robustness of query-agnostic context reconstruction to make compressed caches that work for any input. The experiment would adapt the fast closed-form attention-matching method and layer in a lightweight context reconstruction step inspired by KVzip, hypothesizing better performance in multi-query or conversational settings.

Research Question: Can a hybrid approach that fuses attention-matching compaction with query-agnostic context reconstruction yield more universally effective KV caches for LLM inference across diverse prompt distributions?

Hypothesis: Integrating context reconstruction, as in KVzip, with attention-matching compaction will mitigate quality loss in multi-query and long-conversation scenarios, compared to attention-matching alone.

Experiment Plan: Implement a two-stage KV compaction: first apply fast attention-matching to create a compact latent cache, then reconstruct or augment this cache using query-agnostic scoring (e.g., reconstruct low-importance tokens for broad coverage, as in KVzip). Use datasets with diverse prompt structures (e.g., multi-turn dialogues, retrieval-augmented QA). Measure per-query accuracy, latency, and memory usage, benchmarking against pure attention-matching, KVzip, and standard full-cache baselines. Analyze robustness, especially under high cache reuse and prompt diversity.

References:

• Zweiger, A., Fu, X., Guo, H., & Kim, Y. (2026). Fast KV Compaction via Attention Matching.
• Kim, J.-H., Kim, J., Kwon, S., Lee, J. W., Yun, S., & Song, H. O. (2025). KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. arXiv.org.