Measuring the "Durability" of Methods in AI Research

I’m thinking about the "shelf life" of AI research. We are seeing a flood of papers from 2023 and 2024 that risk becoming instantly outdated the moment a stronger LLM is released. It feels like we have a relevancy crisis: we have thousands of papers that are technically reproducible, but we don't know if the methods they propose are even worth running anymore.

So, that made me wonder: can we measure "durability" of a paper? Maybe we could build a tool that automatically re-runs a paper's method (or maybe simply takes it from their report) and compares it against a SOTA model with a standard prompt. If the newer LLMs outperform the specialized method, the paper is "flagged" as deprecated; if the method holds up, it is marked as "durable". Of course with warnings that maybe a paper's method can be obsolete (performance-wise) but offers durable insights or efficiency gains. But, I think, if being formulated carefully, this would really help filter out "zombie papers", methods that are technically reproducible but practically already dead.

Disclaimer: I’m not dismissing the scientific value of older papers. Every paper contributes to the field's knowledge. This is strictly about distinguishing between methods that remain useful tools versus those that have become historical references.