Failure-Driven RL: Diagnosing and Correcting Multi-Answer Model Pathologies

TL;DR: Instead of just celebrating when your model gets it right, what if you focused on when it gets it weirdly wrong—like giving multiple answers that all miss the mark? Let’s look for these failures and train models to recognize and fix them.

Research Question: Can systematically analyzing and leveraging multi-answer RL model failure cases—such as when all generated answers are incorrect or implausibly distributed—lead to more robust distributional reasoning?

Hypothesis: By explicitly surfacing and penalizing pathological output distributions during RL (e.g., all answers wrong, or mode collapse despite diversity rewards), models will develop better self-awareness and more faithfully reflect true uncertainty, especially in ambiguous or OOD settings.

Experiment Plan: - Extend the multi-answer RL framework to log and cluster failure cases as described in CRAFT (Liu et al., 2026) and FT-CPG (Zhang et al., 2025).

• Develop auxiliary “failure detectors” (small classifiers or heuristic rules) that flag suspiciously confident or implausible answer sets.
• Incorporate a corrective RL signal: when a batch is flagged as pathological, apply tailored penalties or trigger additional exploration.
• Evaluate on ambiguous QA and medical diagnosis tasks, measuring not just accuracy/diversity but also reduction in pathological error rates.

References:

• Liu, Y., Zhang, W., Guo, D., Cao, C., Yuan, F., Sun, Q., Liu, Y., Hong, J. B., & Ma, Z. (2026). CRAFT: Calibrated Reasoning with Answer-Faithful Traces via Reinforcement Learning for Multi-Hop Question Answering.
• Zhang, P., Hua, Z., Qiu, Q., & Ding, J. (2025). FT-CPG: Learning Central Pattern Generators for Fault-Tolerant Quadruped Locomotion Under Multi-Joint Failures. IEEE Robotics and Automation Letters.