From Flat to Fractal: Modeling Multi-Token Branching Factor Dynamics for Diversity-Aware Language Generation

This research builds on the insight that alignment compresses the generative possibilities of LLMs by reducing token-level branching factor (BF), especially early in generation. It proposes redefining BF as a multi-token, fractal-inspired metric that captures how BF fluctuates and interacts over rolling spans of tokens, rather than as a flat per-token scalar. The approach models inter-token interaction effects to quantify how certain token sequences lock in low-diversity paths or restore diversity, even in strongly aligned models. By diagnosing diversity bottlenecks and fractal recoveries, it aims to develop new decoding strategies such as adaptive sampling and branching mechanisms that promote creative recombination across spans. The framework is designed for real-time integration with decoders, enabling adaptive, feedback-driven generation that controls diversity per span rather than per token. This multi-scale, span-aware perspective addresses the temporal evolution and interaction of diversity in generation, with potential impact on training alignment objectives and decoding methods to balance safety, alignment, and expressive creativity in applications like storytelling, code generation, and reasoning.

References:

1. Creativity Has Left the Chat: The Price of Debiasing Language Models. Behnam Mohammadi (2024). arXiv.org.
2. Generating high-quality and diverse synthetic datasets with large language models: A survey. Abinandaraj Rajendran (2025). World Journal of Advanced Engineering Technology and Sciences.
3. Semantic uncertainty in advanced decoding methods for LLM generation. Darius Foodeei, Simin Fan, Martin Jaggi (2025). arXiv.org.
4. Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation. Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, M. Aßenmacher (2024). Conference on Empirical Methods in Natural Language Processing.
5. Avoidance Decoding for Diverse Multi-Branch Story Generation. Kyeongman Park, Nakyeong Yang, Kyomin Jung (2025). arXiv.org.
6. LLM Probability Concentration: How Alignment Shrinks the Generative Horizon. Chenghao Yang, Ari Holtzman (2025).