Meta-Active Selection: Learning to Select Example Selection Strategies

The literature (e.g., Zhang et al., 2022; Perlitz et al., 2023) shows that no single active selection strategy consistently outperforms others across tasks or domains; sometimes random selection works surprisingly well, sometimes uncertainty or diversity is better. This suggests a meta-learning opportunity: build a controller model that, given a task and a stream of feedback about the impact of different selection strategies (e.g., uncertainty, representativeness, adversarial, curriculum-based), learns to actively switch or blend strategies to maximize ICL performance on the fly. The controller could use reinforcement learning or bandit algorithms to optimize its choices, making the example selection process itself adaptive and context-sensitive. This “meta-active” approach acknowledges the heterogeneity of tasks and domains, aiming to automate the process of strategy selection rather than fixating on a single universal method. It could be a game-changer for scaling in-context learning across diverse settings.

References:

1. Active Learning for Natural Language Generation. Yotam Perlitz, Ariel Gera, Michal Shmueli-Scheuer, D. Sheinwald, N. Slonim, L. Ein-Dor (2023). Conference on Empirical Methods in Natural Language Processing.
2. Active Example Selection for In-Context Learning. Yiming Zhang, Shi Feng, Chenhao Tan (2022). Conference on Empirical Methods in Natural Language Processing.