Feedback-Loops in Prompt Optimization: Modeling the Impact of Iterative Human and Automated Critique

Several recent frameworks—like PromptWizard (Agarwal et al., 2024) and PROMST (Chen et al., 2024)—use feedback to optimize prompts, but there’s little systematic study on how different feedback loops (human, automated, or hybrid) affect the evolution and quality of prompts. This research would experimentally compare various feedback paradigms (e.g., LLM self-critique, human review, preference alignment, or multi-agent collaboration as in MALTA) across tasks, measuring not just output quality but the diversity, robustness, and efficiency of the prompt optimization process. The key innovation is modeling the feedback process itself: How does the nature, timing, and source of feedback shift the search landscape of prompt engineering? Insights could lead to new meta-prompting strategies or “feedback-aware” prompt design tools that accelerate convergence to high-performing prompts.

References:

1. PromptWizard: Task-Aware Prompt Optimization Framework. Eshaan Agarwal, Joykirat Singh, Vivek Dani, Raghav Magazine, Tanuja Ganu, A. Nambi (2024).
2. PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling. Yongchao Chen, Jacob Arkin, Yilun Hao, Yang Zhang, Nicholas Roy, Chuchu Fan (2024). Conference on Empirical Methods in Natural Language Processing.
3. PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Preference Alignment. Yongchao Chen, Jacob Arkin, Yilun Hao, Yang Zhang, Nicholas Roy, Chuchu Fan (2024). arXiv.org.
4. MALTA: An Automated CGRA Design Framework. Zesong Jiang, Yuqi Sun, Qing Zhong, Mahathi Krishna, Deepak Patil, Cheng Tan, Sriram Krishnamoorthy, Jeff Zhang (2025). arXiv.org.