Uncertainty-Aware Multi-Answer RL for Multimodal Reasoning

TL;DR: What if a language model could not only suggest multiple answers, but also say how sure it is about each—across both text and images? To test this, we’ll train a model using RL to generate diverse answer sets and attach calibrated uncertainty estimates for each, leveraging both textual and visual cues.

Research Question: How can we jointly model answer diversity and uncertainty in multimodal language models using reinforcement learning, enabling the model to deliver confidence-calibrated multi-modal hypotheses in a single pass?

Hypothesis: By explicitly incorporating uncertainty quantification into the RL objective, trained models will produce more informative and calibrated answer sets in multimodal reasoning tasks, outperforming approaches that separate answer generation from uncertainty estimation.

Experiment Plan: - Build upon a multimodal SLM setup (Lin et al., 2025), adapting the RL objective to not only encourage diverse answer sets (as in the original paper) but also maximize calibration metrics (e.g., ECE, Brier score) on predicted confidences.

• Leverage probabilistic embeddings (Venkataramanan et al., 2025) as an auxiliary head to produce uncertainty estimates for each generated answer.
• Evaluate on image-question and mixed-modality datasets, measuring accuracy, answer diversity, and uncertainty calibration.
• Compare to baseline multi-answer RL without uncertainty and to post-hoc calibration methods.

References:

• Lin, Y.-C., Sharma, S., Manikandan, H., Kumar, J., King, T., & Zheng, J. (2025). Scaling Multimodal Search and Recommendation with Small Language Models via Upside-Down Reinforcement Learning. 2025 IEEE International Conference on Data Mining Workshops (ICDMW).
• Venkataramanan, A., Bodesheim, P., & Denzler, J. (2025). Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models. Conference on Uncertainty in Artificial Intelligence.