IndexCache Meets Pruning: Joint Cross-Layer Index Reuse and Sparse Model Compression

TL;DR: What if we combined index reuse with neural network pruning, slashing both indexer computations and actual model weights at the same time? The initial experiment would jointly optimize pruning masks and indexer schedules to see if compound efficiency gains are possible.

Research Question: How can cross-layer index reuse (IndexCache) be synergistically combined with structured/unstructured pruning to maximize both model sparsity and inference efficiency for large language models?

Hypothesis: By aligning the layers or heads with low indexer redundancy (good for reuse) and high redundancy in weights (good for pruning), we can co-optimize both forms of sparsity, compounding memory and compute savings without significant accuracy loss.

Experiment Plan: - Analyze the correlation between indexer redundancy and weight redundancy (e.g., by measuring activation and gradient sparsity) across layers and heads.

• Design a joint optimization procedure (training-aware or training-free) that selects indexer placements and pruning masks together.
• Test on LLMs with sparse attention, measuring wall-clock speedup, memory reduction, and accuracy on long-context tasks.
• Compare to baselines using only IndexCache or only pruning.

References:

• Bai, Y., Dong, Q., Jiang, T., Lv, X., Du, Z., Zeng, A., Tang, J., & Li, J. (2026). IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse.
• Fan, R., Yu, X., Dong, P., Li, Z., Gong, G., Wang, Q., Wang, W., & Chu, X.-X. (2025). SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs. European Conference on Computer Systems.