Generative Counterfactual Training: Teaching Language Models to "See" What’s Missing

Building directly on the findings of AbsenceBench, which shows that LLMs are good at finding "what’s there" (NIAH) but stymied by "what’s missing," this idea proposes a fundamentally different training approach: Instead of only asking models to find deleted spans given source and edited contexts, we train them to actively generate and justify plausible missing content—even in the absence of explicit cues. Inspired by counterfactual reasoning and adversarial training from anomaly detection, the approach involves synthesizing plausible deletions (e.g., typical poetry stanzas or expected code logic) and teaching models to both identify gaps and rationalize what could have filled them. The research agenda includes data augmentation via retrieval-augmented generation to create plausible missing content, counterfactual training to output likely content that might have existed, rationalized absence detection to justify omissions and their impacts, and attention supervision to guide models to search for expected structures or themes. This approach moves beyond token-wise comparison, models implied expectations, is inherently adversarial, and cross-pollinates with anomaly detection methodologies. Potential impacts include improved fact/logic checking, expectation-oriented criticism, detection of omitted steps in code reviews, subtle poetry structure lapses, and advances in explainable AI by clarifying why absences matter.