Dynamic Error-Driven Example Selection: Learning from Model Surprises

While papers like STAYKATE (Zhu et al., 2024) and AdaICL (Mavromatis et al., 2023) explore hybrid and uncertainty-based selection, they typically focus on static notions of representativeness, diversity, or model uncertainty. However, Monte Carlo Sampling for Analyzing In-Context Examples (Schoch & Ji, 2025) reveals unexpected performance degradations and outliers, suggesting that areas where the model “surprises” us—either by failing or succeeding unexpectedly—hold untapped potential for learning. This research would propose a feedback loop where, after each batch of in-context predictions, the system measures the delta between expected and actual model outputs (using, for instance, confidence, entropy, or calibration error). It then actively samples future examples from regions of the input space where these “surprises” are most prevalent. This approach not only targets the model’s blindspots but also embraces the brittleness and instability highlighted by Zhang et al. (2022), potentially leading to more robust and generalizable in-context learning. The novelty lies in operationalizing “unexpectedness” as a selection signal, rather than just uncertainty or diversity, and in embracing performance anomalies as learning opportunities rather than discarding them as negative results.

References:

1. STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach - A Case Study on Science Domains. Chencheng Zhu, Kazutaka Shimada, Tomoki Taniguchi, Tomoko Ohkuma (2024). arXiv.org.
2. Monte Carlo Sampling for Analyzing In-Context Examples. S. Schoch, Yangfeng Ji (2025). The Sixth Workshop on Insights from Negative Results in NLP.
3. Active Example Selection for In-Context Learning. Yiming Zhang, Shi Feng, Chenhao Tan (2022). Conference on Empirical Methods in Natural Language Processing.
4. Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection. Costas Mavromatis, Balasubramaniam Srinivasan, Zhengyuan Shen, Jiani Zhang, H. Rangwala, Christos Faloutsos, G. Karypis (2023). arXiv.org.