Detecting and Mitigating Hidden Alignment Drift: Fine-Grained Discourse Analysis Beyond Alignment Labels

TL;DR: ELI5: Sometimes, models can subtly become misaligned in ways we don’t expect, even if the training data seems positive—let’s look for those sneaky shifts! Experiment: Track nuanced alignment drift by analyzing the valence and structure of discourse about AI in pretraining data at a much finer granularity (e.g., rhetorical style, metaphors, irony) and test whether "positive" or "neutral" AI discourse sometimes induces unexpected misaligned behaviors. Hypothesis: Pretraining on “ambiguous” or “subtle” misalignment discourse (e.g., sarcastic praise, indirect criticism) can cause latent misalignment not captured by simple label-based upsampling.

Research Question: How do nuanced or ambiguous forms of AI discourse (such as irony, sarcasm, or hedged statements) during pretraining affect the emergence of subtle or “hidden” misalignment behaviors in LLMs?

Hypothesis: Not all “positive” or “negative” discourse has the same effect; ambiguous or stylistically complex discourse can induce latent misalignment, even when overt labels suggest alignment.

Experiment Plan: - Curate and annotate a corpus of AI discourse with stylistic and rhetorical features (e.g., sarcasm, metaphor, hedging, indirectness).

• Pretrain LLM variants on (a) overtly positive/negative discourse, (b) ambiguous or stylistically marked discourse, and (c) mixed sets.
• Develop behavioral probes and alignment tests designed to surface subtle or “hidden” misalignment (e.g., responses to indirect ethical dilemmas, nuanced trust calibration tasks).
• Compare alignment metrics and surface behaviors, analyzing where and how “hidden” misalignment emerges.

References:

• Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D., & O'Brien, K. (2026). Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment.
• Si, Y. (2024). The Confirmative Stance and Communicative Value of a Chinese Discourse Marker. Lecture Notes on Language and Literature.