Human-in-the-Loop Faithfulness Adjudication: Crowdsourced and Expert-Guided Correction of LLM Legal Summaries

FABLES (Kim et al. 2024) and CaseSumm (Heddaya et al. 2024) both highlight the limitations of automated faithfulness evaluation and the persistent gap between LLM raters and human judgment. This idea takes it a step further by proposing a participatory, human-in-the-loop framework: after LLMs generate a legal summary, both legal experts and trained crowdsourced annotators (who have read the case/book) flag unfaithful or misleading claims, suggest corrections, and annotate the nature of the error (e.g., omission, hallucination, misattribution). These annotations and corrections are then used both to improve evaluation metrics and to fine-tune models (akin to the RLHF paradigm, but with domain-specific interventions). By closing the loop between users and model improvement, this approach could rapidly bootstrap more faithful, trustworthy models and create new datasets that capture nuanced legal faithfulness errors—something current automated pipelines cannot achieve.

References:

1. FABLES: Evaluating faithfulness and content selection in book-length summarization. Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, Mohit Iyyer (2024). arXiv.org.
2. CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions. Mourad Heddaya, Kyle MacMillan, Anup Malani, Hongyuan Mei, Chenhao Tan (2024). North American Chapter of the Association for Computational Linguistics.