'Most Underrated' as a Novelty Metric

by Ari Holtzmanabout 1 month ago
1

LLMs are bad at novelty so far. It is not entirely clear why.

One key thing for LLMs to have enough taste that they could potentially be novel: name the most underrated thing in a category X. If they name things that are widely-believed to be underrated, this doesn't count.

Of course, these lines are fuzzy, but my limited experience suggests LLMs are losing so badly we could use this as a diagnostic for now: LLMs name the 'obvious' underrated if you go to the r/X subreddit for some value of X.

If LLMs can't reason about what's underrated for a deeper reason than 'people said it was underrated' I think they'll have trouble truly innovating.

llmsnoveltymetricsImplemented:https://github.com/Hypogenic-AI/underrated-metric-nlp-0ca5-claudeImplemented:https://github.com/Hypogenic-AI/underrated-metric-nlp-f82f-codexImplemented:https://github.com/Hypogenic-AI/underrated-metric-nlp-51aa-gemini
Chat

If you are inspired by this idea, you can reach out to the authors for collaboration or cite it:

@misc{holtzman-most-underrated-as-2026,
  author = {Holtzman, Ari},
  title = {'Most Underrated' as a Novelty Metric},
  year = {2026},
  url = {https://hypogenic.ai/ideahub/idea/UmbYQyYnr8fjZJ7GxXam}
}

Comments (0)

Please sign in to comment on this idea.

No comments yet. Be the first to share your thoughts!