Out of order memorization.

3

An idea I've been thinking about for a long time is very simple: If a few documents a language model sees contain AB, and a few more documents a language model sees contain BC, and a few more documents contain CD, then can a language model implicitly memorize ABCDEFGH, etc.? If so, show me. If not, why not?

llms memorization finetuning pretraining ai safety safety

Chat

If you are inspired by this idea, you can reach out to the authors for collaboration or cite it:

@misc{holtzman-out-of-order-2025,
  author = {Holtzman, Ari},
  title = {Out of order memorization.},
  year = {2025},
  url = {https://hypogenic.ai/ideahub/idea/i0EfeNedpgWZsJZFwjDH}
}

Comments (0)

Please sign in to comment on this idea.

No comments yet. Be the first to share your thoughts!