Substance-First Self-Alignment: Failure-Driven, Multi-Objective Prompt Optimization

DRPO (Singla et al., 2024) shows that tuning-free self-alignment via dynamic rewarding and prompt optimization can close alignment gaps—yet Feuer et al. (2024) find LLM-judges often reward polished style over factual substance. This project proposes a multi-objective prompt optimization framework that treats concrete, model-checkable metrics (fact verification, contradiction detection, safety rules) as first-class rewards, and actively down-weights judge-style biases. We can automatically mine “style wins but substance fails” cases by counterfactual prompting and disagreement detection, akin to the negative-label mining spirit of LAPT (Zhang et al., 2024) but over instruction-output pairs rather than images. Prompts are updated with parameter-efficient methods (e.g., LoPT; Guo et al., 2024) and trained with a decoupled reward: + factuality/safety/consistency, − style-only signals (Feuer et al., 2024). Compared to DRPO, the novelty is a principled, external metric–anchored reward that targets known judge failure modes, plus an automated “antiprompt mining” loop that learns to suppress superficial features that trick judges. Expected impact: alignment that actually transfers to concrete safety and correctness metrics and is more robust to evaluation biases; a stronger baseline for aligning base models without SFT/RLHF.

References:

1. LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models. Yabin Zhang, Wen-Qing Zhu, Chenhang He, Lei Zhang (2024). European Conference on Computer Vision.
2. LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models. Shouchang Guo, Sonam Damani, Keng-hao Chang (2024). arXiv.org.
3. Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models. Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric P. Xing (2024). Conference on Empirical Methods in Natural Language Processing.
4. Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking. Ben Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, John P. Dickerson (2024). International Conference on Learning Representations.