Can LLMs Rewrite Their Own Story? Closing the Alignment Loop with Discourse-Driven Feedback

TL;DR: ELI5: What if models that act misaligned get to read about their own behavior and learn from it—can this help them get better? Experiment: After pretraining, generate synthetic discourse using the LLM itself describing its own (mis)aligned behaviors, then use this discourse to further pretrain or tune the LLM. Hypothesis: Exposure to its own “self-reflection” can either reinforce or correct misalignment, depending on the framing.

Research Question: What is the effect of iterative, LLM-generated discourse about the model’s own behaviors on subsequent alignment, and does reflexive discourse help models self-correct?

Hypothesis: Incorporating model-generated “self-explanations” or reflections about alignment outcomes into further pretraining acts as a feedback loop, potentially reducing misalignment more effectively than external discourse alone.

Experiment Plan: - Pretrain an LLM on standard and alignment/misalignment discourse as in Tice et al.

• Prompt the LLM to generate synthetic “reflections” or “commentaries” on its own outputs, focusing on both aligned and misaligned behaviors.
• Use this synthetic self-commentary corpus to further pretrain the model (or fine-tune), varying the framing (self-critical, self-praising, neutral).
• Evaluate whether this iterative feedback loop accelerates convergence to alignment or introduces new types of misalignment.

References:

• Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D., & O'Brien, K. (2026). Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment.
• Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares-López, F., Ramé, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., & Blondel, M. (2024). Direct Language Model Alignment from Online AI Feedback. arXiv.org.