Qualitative User-in-the-Loop Example Selection for Domain-Specific In-Context Learning

Most approaches (e.g., STAYKATE, AdaICL) are fully automated and rely on model-internal signals. However, Out of Sesame Street (Nunes et al., 2024) highlights the challenges of domain-specific NER in legal Portuguese, where expert insight into which examples are “representative” or “edge cases” could be invaluable. This research proposes a semi-automated, user-in-the-loop pipeline: after an initial automated selection, domain experts qualitatively annotate or tag examples as “typical,” “ambiguous,” or “critical errors.” The system then actively selects subsequent examples to probe these boundaries or test the model on expert-identified failure modes. This approach leverages open-ended qualitative exploration, giving more weight to human intuition and domain knowledge than current fully automated systems. Such a hybrid strategy could be particularly valuable in low-resource or specialized settings where subtle context matters, and could help surface rare but critical phenomena that models might otherwise miss.

References:

1. STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach - A Case Study on Science Domains. Chencheng Zhu, Kazutaka Shimada, Tomoki Taniguchi, Tomoko Ohkuma (2024). arXiv.org.
2. Out of Sesame Street: A Study of Portuguese Legal Named Entity Recognition Through In-Context Learning. Rafael Oleques Nunes, A. Spritzer, Carla Maria Dal Sasso Freitas, D. G. Balreira (2024). International Conference on Enterprise Information Systems.