Emotion- and Context-Aware Multimodal Intent Detection for LLM Safety

TL;DR: What if LLMs could “hear” the emotion in your voice or “see” your facial expressions to better judge your intent? By incorporating audio, facial, and behavioral cues into intent detection, LLMs might spot malicious or manipulative intent hidden behind emotionally charged or ambiguous language. A pilot study could train a multimodal LLM on datasets of spoken and written adversarial prompts, annotated for emotion and intent.

Research Question: Can integrating multimodal user signals—such as emotional tone and facial expressions—into LLM safety mechanisms improve detection of concealed or contextually ambiguous malicious intent?

Hypothesis: Multimodal intent recognition will flag more adversarial attempts that rely on emotional manipulation or ambiguity than text-only systems.

Experiment Plan: - Collect or simulate a dataset of multimodal prompts (text, audio, facial cues) including emotional framing attacks (Feng et al., 2025).

• Train a multimodal LLM (cf. Bian et al., 2023; Jeniffer et al., 2025) to classify intent, leveraging both linguistic and paralinguistic features.
• Evaluate against standard LLMs and emotional-agnostic detectors on adversarial as well as benign conversations.
• Measure precision, recall, and robustness to new attack types.

References:

• Feng, B.-H., Liu, C.-F., Li Liang, Y.-H., Yang, C.-K., Fu, S.-W., Chen, Z., Lu, K.-H., Huang, S.-F., Wang, Y.-C. F., Chen, Y.-N., & Lee, H.-y. (2025). Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations. arXiv.org.
• Bian, Y., Küster, D., Liu, H., & Krumhuber, E. G. (2023). Understanding Naturalistic Facial Expressions with Deep Learning and Multimodal Large Language Models. Italian National Conference on Sensors.
• Jeniffer, T. J., Swetha, M., Raghuvaran, E., D. R, & S. R. (2025). Enhancing Sentiment Analysis with Multimodal Large Language Models. 2025 6th International Conference for Emerging Technology (INCET).
• Ahmed Mohamed Hussain, Salahuddin Salahuddin, P. Papadimitratos. (2025). Beyond Context: Large Language Models Failure to Grasp Users Intent.