From Anomaly to Absence: Adapting Log and Network Anomaly Detection for Missing Information in Natural Language

TL;DR: What if we borrow anomaly detection tricks from system logs and network monitoring, where "missing events" often signal trouble, and apply them to LLM-based document analysis? A first attempt can use semantic anomaly detection methods (like LogLLM or BERT-based anomaly detectors) to flag instances of missing information.

Research Question: Can techniques from log-based and network anomaly detection be adapted to identify missing (rather than anomalous or unexpected) information in natural language documents?

Hypothesis: Methods designed to detect anomalies in structured/unstructured log sequences—especially those leveraging semantic embeddings and sequence regularity—can be repurposed for missing information detection, outperforming LLM-only baselines.

Experiment Plan: - Methodology: Apply or adapt BERT/LLM-based anomaly detectors (e.g., LogLLM, Lookback Lens) to AbsenceBench domains.

• Data: Use AbsenceBench and augment with additional document types (e.g., news articles with redacted content).
• Metrics: Standard F1-score plus anomaly detection metrics (precision/recall for absences).
• Comparison: Compare against vanilla LLMs and previous hallucination detectors.
• Expected Outcome: Anomaly detection frameworks should flag missing content with higher recall, especially in structured domains.

References:

• Zhou, Y., Chen, Y., Rao, X., Zhou, Y., Li, Y., & Hu, C. (2024). Leveraging Large Language Models and BERT for Log Parsing and Anomaly Detection. Mathematics.
• Guan, W., Cao, J., Qian, S., & Gao, J. (2024). LogLLM: Log-based Anomaly Detection Using Large Language Models. arXiv.org.