Multi-Modal Checklist Evaluation: Integrating Structured Data, Voice, and Text for Comprehensive Note Assessment

Most current approaches (e.g., Ben Abacha et al., 2023; Li et al., 2024) focus solely on the generated text, but valuable clinical information may reside in structured EHR fields or even the original audio (as in Relisten, Wang et al., 2023). This research proposes a multi-modal evaluation framework, where checklist fulfillment is assessed by cross-referencing the generated note with relevant EHR data fields (e.g., lab results, vital signs) and, optionally, the original conversation audio using state-of-the-art speech recognition. The system could, for example, flag discrepancies where a lab value mentioned in the conversation or EHR is omitted or misrepresented in the generated note. This approach challenges the assumption that text alone suffices for evaluation and could lead to higher accuracy in detecting omissions, hallucinations, or context loss, making automated evaluation more robust and clinically meaningful.

References:

1. An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters. Asma Ben Abacha, Wen-wai Yim, Yadan Fan, Thomas Lin (2023). Conference of the European Chapter of the Association for Computational Linguistics.
2. Improving Clinical Note Generation from Complex Doctor-Patient Conversation. Yizhan Li, Sifan Wu, Christopher W. Smith, Thomas Lo, Bang Liu (2024). Pacific-Asia Conference on Knowledge Discovery and Data Mining.
3. Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation. Hanyin Wang, Chufan Gao, Bolun Liu, Qiping Xu, Guleid Hussein, Mohamad El Labban, Kingsley Iheasirim, H. Korsapati, Chuck Outcalt, Jimeng Sun (2024). Annual Meeting of the Association for Computational Linguistics.