Elhage et al. 2021 and Olsson et al. 2021 famously discovered and described induction heads. I've seen surprisingly little about them since. One thing I've always wondered—they found induction heads through copying on random data. Are there induction heads (or something like them) that only work on data that looks more like the training set?
If you are inspired by this idea, you can reach out to the authors for collaboration or cite it:
@misc{holtzman-naturalistic-induction-heads-2026,
author = {Holtzman, Ari},
title = {Naturalistic Induction Heads},
year = {2026},
url = {https://hypogenic.ai/ideahub/idea/6nfMPtf7OaP6HZShqqog}
}Please sign in to comment on this idea.
No comments yet. Be the first to share your thoughts!