Probing the Limits: When Does Index Reuse Fail? A Fine-Grained Error Profiling and Robustness Study

TL;DR: Let’s systematically find and analyze the rare cases where index reuse in IndexCache causes unexpected accuracy drops, to guide more robust or adaptive methods. The first study would profile error spikes across tasks, layers, and inputs.

Research Question: What are the failure modes and input/task characteristics where cross-layer index reuse in IndexCache leads to significant degradation, and how can we design mechanisms to detect or mitigate these cases in production?

Hypothesis: While IndexCache works well on average, there exist edge cases (e.g., abrupt topic shifts, adversarial prompts, or specific linguistic phenomena) where index reuse fails, causing accuracy cliffs; identifying and understanding these cases can inform fallback strategies or adaptive reuse.

Experiment Plan: - Collect a diverse set of long-context benchmarks, including adversarial and out-of-distribution examples.

• Instrument the model to log per-layer, per-input changes in attention distributions and downstream metrics (e.g., loss, perplexity, specific task scores) when index reuse is applied.
• Identify patterns or clusters of failure (e.g., sudden divergence in top-k indices, rare token bursts).
• Propose and evaluate runtime detection heuristics (e.g., abrupt index divergence, confidence thresholds) that trigger re-indexing or revert to full attention when needed.

References:

• Bai, Y., Dong, Q., Jiang, T., Lv, X., Du, Z., Zeng, A., Tang, J., & Li, J. (2026). IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse.
• Li, H., Li, Z., Bai, Z., & Mitra, T. (2024). ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing. International Symposium on High-Performance Computer Architecture.