Hardware-Aware IndexCache: Co-Designing Index Reuse with Next-Gen Accelerators

TL;DR: Let’s design new hardware (or adapt existing ones) that natively accelerates cross-layer index reuse, possibly by caching and reusing indices in on-chip memory. An experiment could prototype a hardware simulator or FPGA implementation that measures speed and energy gains from such co-design.

Research Question: How can emerging hardware architectures (e.g., Processing-in-Memory, neuromorphic chips) be co-designed with IndexCache-style cross-layer index reuse to optimize memory access patterns and throughput for sparse attention?

Hypothesis: Hardware that exposes fast, low-latency local memory or supports efficient index sharing across layers could further amplify the benefits of IndexCache, especially in bandwidth-constrained or energy-limited settings.

Experiment Plan: - Model memory and compute patterns of IndexCache-enabled sparse attention on current accelerators.

• Propose architectural modifications (e.g., dedicated index reuse buffers, in-situ index computation units) tailored to IndexCache.
• Simulate or prototype (using FPGA or cycle-accurate simulators) the performance and energy benefits.
• Compare to standard GPU or CPU implementations, focusing on long-sequence inference.

References:

• Bai, Y., Dong, Q., Jiang, T., Lv, X., Du, Z., Zeng, A., Tang, J., & Li, J. (2026). IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse.
• Li, H., Li, Z., Bai, Z., & Mitra, T. (2024). ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing. International Symposium on High-Performance Computer Architecture.