How many characters can a model keep track of?

by Ari Holtzman5 months ago
3

Some stories have a lot of characters—think War and Peace or the Tale of Genji. Given a new story, in-context, how many different characters can a model keep track of? And can we use the points at which the model struggles to help us figure out what mechanism it uses to keep track of different characters and their attributes? How good are models at this anyway?

LLMmechinterpstoriesvariable bindingImplemented:https://github.com/Hypogenic-AI/model-character-limit-claudeImplemented:https://github.com/Hypogenic-AI/model-character-limit-codexImplemented:https://github.com/Hypogenic-AI/model-character-limit-gemini
Chat

If you are inspired by this idea, you can reach out to the authors for collaboration or cite it:

@misc{holtzman-how-many-characters-2025,
  author = {Holtzman, Ari},
  title = {How many characters can a model keep track of?},
  year = {2025},
  url = {https://hypogenic.ai/ideahub/idea/ihOvpOiQwTAAjxxdtWuv}
}

Comments (1)

Please sign in to comment on this idea.

Mark Edwards5 months ago

Interesting question, normally we would remember characters with relevance to the story/arc and the protagonist, so sort of a priority cluster mechanism I suppose. I assume a model would remember characters based on their sequential frequency? maybe, would like to see how the correlation mechanism relates characters and if the model's 'failure points' occur when a low-frequency character performs a high-relevance action.

0