Counterfactual Summarization: Stress-Testing Faithfulness via Synthetic Document Reordering and Perturbation

Inspired by Wan et al. (2024) and HERA (Li et al. 2025), both of which suggest that document order and context structure significantly impact summarization faithfulness, this research proposes a systematic framework for counterfactual testing: take legal documents and generate "synthetic" versions with shuffled sections, omitted arguments, or reordered facts. Summaries generated from these perturbed documents are then compared for fidelity to the true legal facts and logic. This approach extends prior work by providing a rigorous, adversarial testbed for faithfulness—uncovering not just average-case performance but worst-case vulnerabilities (e.g., does the model hallucinate when the timeline is scrambled? Does it invent facts to bridge incoherence?). The resulting dataset and evaluation suite could drive the next generation of robust, context-invariant legal summarizers.

References:

1. On Positional Bias of Faithfulness for Long-form Summarization. David Wan, Jesse Vig, Mohit Bansal, Shafiq R. Joty (2024). North American Chapter of the Association for Computational Linguistics.
2. HERA: Improving Long Document Summarization using Large Language Models with Context Packaging and Reordering. Taiji Li, Hao Chen, Fei Yu, Yin Zhang (2025). arXiv.org.