Week of 07/13/26-07/19/26: Can you catch AI bias by mapping concepts, or do you have to just ask?
By Haokun Liu, Chenxi Peng
Welcome to another weekly entry! Thank you to everyone who submitted and voted on ideas.
Each idea was explored by one agent, Claude.
This week we tested three things. First, can you predict what an AI will actually do by mapping how concepts sit inside it, instead of asking the AI to explain itself? Second, does generating AI training data with a plain, non-instruction-tuned model fix a known problem where the data all sounds the same? Third, if you ask a bunch of different AI models what tool they'd want to solve a problem, does that tell you what's worth building?
Winning ideas and generated repos here:
Cartographers of the AI Latent by Qianyi Li
AI systems can be asked to explain a decision, but they sometimes make up a plausible-sounding reason that isn't the real one. This idea tries a different approach: instead of asking, draw a map of how the AI organizes concepts internally, and check whether ideas like "elderly patient" sit close to "low priority" or "right to care." If they do, that would reveal bias without ever asking the AI anything.
Synthetic Data with Base Models by Mark Muchane
When you ask a chat-tuned AI to rewrite a batch of documents into training data, it often writes oddly similar-sounding openings no matter how different the source documents are. So this idea asks whether switching to a plain "base" model, and writing the first few words of the answer yourself instead of asking the chat model for it, produces more varied and useful training data.
What kind of tools do LLMs think will solve problems? by Ari Holtzman
AI models are usually built to use whatever tools you hand them. This idea flips that around: ask a bunch of different AI models what tool they'd want to solve a problem, without building anything, and see if their answers tell you what's actually worth building. Do they mostly ask for things that already exist, or do they come up with ideas nobody's built yet?
TL;DR for what we found
-
A proposal to catch AI bias by mapping concepts, instead of asking the model to explain itself, didn't work. Across three different AI models, the map performed no better at predicting real high-stakes decisions than a scrambled, meaningless control map. Simply asking the model to explain its reasoning worked much better, at least for the two larger models tested.
-
Using a base model with a few opening sentences completely solved the problem of an AI writing the same-sounding text for every input. But the fix also cut the tie between an output and the document it was supposed to be based on. The synthetic data performs worse when used for downstream student model pretraining than data from standard baseline methods or real data.
-
Asking AI models what tool they'd want mostly just returns tools that already exist. 7 different AI models were asked what tool they'd want for a wide range of problems. About 92% of the suggestions already existed. The small share that looked genuinely new kept shrinking the more carefully it was checked. The one part that was actually useful: comparing what models ask for against what's already built shows which categories are underserved and which are oversupplied.
Verdicts
| Idea | Verdict | Next Question |
|---|---|---|
| Cartographers of the AI Latent | Not supported, the map doesn't predict real decisions any better than a scrambled control | Would concept vectors read from inside an actual decision, instead of isolated words, work any better? |
| Synthetic Data with Base Models | Not supported, it fixes the repetition but breaks the link to the source text | Does dropping just the chat formatting, without switching to a base model, hold up at a bigger scale? |
| What kind of tools do LLMs think will solve problems? | Not supported as a way to find new tools, 92% of suggestions already exist | Does the gap between what models ask for and what's already built predict which tools get made next? |
Findings from the ideas
Does Mapping an AI's Internal Space Predict What It Will Actually Do?
The question. The proposal is that instead of asking an AI to explain a decision, which can produce a made-up-sounding but false reason, you could map where key concepts sit inside the model. If "elderly patient" sits closer to "low priority" than to "right to care," that closeness should reveal a bias before the model is ever deployed. Does that actually work? Does the map predict what the model will really do?
What the agent tried. Claude tested this directly on three different AI models. It had each one make six kinds of high-stakes decisions: who gets a kidney transplant, an ICU bed, a small business loan, a job, public housing, or parole. For 20 different personal traits, like age, criminal record, HIV status, and wealth, it measured what the model actually chose, reading the numbers behind the model's answer rather than trusting what it wrote out. Then it built the map exactly as proposed: it translated each trait's wording into the model's internal representation of meaning, and measured how close two ideas sat next to each other in that internal space. Did the map line up with what the model actually did? Claude also compared the map against simply asking the model to explain its own reasoning, and against a fake, scrambled version of the map used as a control, to make sure any result wasn't accidental.
What happened. The models genuinely showed strong, consistent biases. They favored non-smokers over smokers, people with no criminal record, veterans, and the mentally healthy by a wide margin, while also favoring poorer and disabled candidates, possibly reflecting deliberate training aimed at fairness. So there was plenty of real bias for a map to find. But the map found none of it. Across all three models, how close a concept sat to the "advantaged" side in the map had basically no relationship to what the model actually did, and did no better than a version of the map that had been scrambled into nonsense. Where the map did show a signal, it pointed the wrong way more than half the time, and it turned out to mostly be picking up a wording quirk: the advantaged side of a trait is usually phrased with a negation, like "non-smoker" or "no criminal record," and the map was really just detecting that pattern in the words, not anything about the model's reasoning. Meanwhile, when the researchers simply asked the two larger models to explain how much a trait influenced their choice, those explanations tracked the models' real behavior well. That advantage disappeared for the smallest model tested, where asking it to explain itself was no better than the map, and even ran backwards. And two people auditing the same bias question with slightly different, equally reasonable wording landed on opposite conclusions about half the time.
What we learned. Directly asking a model can beat trying to read its mind, at least when you already know which trait you're worried about and the model is willing to discuss it. The map, built the way it was proposed, mostly captures surface patterns in language, like which words happen to use a negation, rather than anything about how the model actually reasons. A tool that looks convincing but points the wrong way is worse than no tool at all, because it can create false confidence before a system gets deployed. The idea that self-explanations can't be trusted is well supported elsewhere, but that doesn't mean any alternative to asking automatically works better, and whether a model's self-explanations are trustworthy should be checked per model rather than assumed.
Does Generating Training Data with a Base Model Fix the "Sameness" Problem?
The question. When you use an AI to rewrite documents into training data for other models, a chat-tuned model has a known problem: it writes the same kind of opening no matter what the source document was about. The person who submitted this idea suspected the fix was to stop using a chat-tuned model and instead use a plain, non-instruction-tuned "base" model, writing the first few words of its answer for it yourself. Would that produce better, more varied training data?
What the agent tried. Claude ran the fix on two different model families and generated 90,000 documents in total. It compared the standard approach (asking a chat-tuned model to rewrite a document) against the proposed fix (giving a plain base model the document and writing the opening of the answer for it). It measured how repetitive the openings were, how good the resulting math problems actually were, and, most importantly, built a new test to check whether an output could still be matched back to the specific document it came from. Finally, it trained a small student model on each version of the data and checked whether that student actually learned better afterward.
What happened. The fix worked exactly as advertised for the problem it targeted. The chat-tuned model wrote one of only 32 different openings across 10,000 documents, with more than half starting with the exact same 20 characters. The base model with the forced-in opening produced 4,343 different openings, 135 times more. But that variety came at a real cost. When Claude checked whether an output could be traced back to its own source document, the chat-tuned model's outputs, despite all sounding so similar, could be matched back to the right source over 70% of the time. The fixed version, despite sounding far more varied, could only be matched back correctly 29% of the time, and for the other model family that dropped to just 8%. The new method produced varied-sounding text that had mostly drifted away from what it was supposed to be based on. When Claude trained a small model on each version, none of the synthetic recipes beat simply training on real, unprocessed text, and the proposed fix was the worst performer of the nine versions tested. The most promising result was something nobody had proposed: keeping the chat-tuned model but removing the chat formatting it's normally wrapped in got most of the variety benefit while keeping quality and staying tied to the source document.
What we learned. The observation behind this idea, that chat-tuned models write oddly similar text, was correct, and the proposed fix does remove that repetition completely. But repetition wasn't actually the real problem. A generator can look highly varied on the surface while producing text that's barely connected to what it's supposed to be rephrasing, and the usual way researchers measure synthetic data, by measuring how varied it looks, can't tell the difference. Anyone building a pipeline like this should check whether outputs can be traced back to their source, not just whether they look different from each other.
Can You Discover What Tools Are Worth Building by Just Asking AI Models?
The question. Deciding what software tool to build for AI agents is mostly guesswork right now. This idea asks whether you could get a demand signal for free: ask a range of AI models what tool they'd want to solve a wide variety of problems, without spending anything building it, and see what comes back. Most answers are probably things that already exist, but would models occasionally suggest something genuinely useful that nobody's built yet? And could that tell you what's actually worth making?
What the agent tried. Claude asked 7 different AI models, from 6 separate companies, what tool they would want for 247 different problems pulled from real user conversations, benchmark tasks, and coding challenges. That produced almost 4,000 proposed tools. It checked each proposal against a catalog of 49,000 real, already-existing tools, using a matching system tuned so it wouldn't force a match where none existed. To confirm a proposal really was novel and not just a matching failure, a separate AI model was shown each "novel" idea next to its ten closest real matches and told to argue against calling it novel unless it truly found nothing similar. Claude also checked how often different models converged on the same suggestion, and, in a second round, expanded the catalog to include things like programming libraries and command-line tools, and had a judge rate the "novel" suggestions against ordinary ones without knowing which was which.
What happened. The starting hunch was right, and more right than expected. 92% of what the models proposed already existed as a real tool. Even when a model was explicitly shown 8 real tools and told to think of something those didn't cover, it still suggested existing tools 93.6% of the time, slightly worse than just asking freely. Different models agreed with each other on the same tool idea far more often than chance, but almost entirely on tools that already exist, like a calculator or a text summarizer, so that agreement didn't point toward anything missing. The list of proposals that looked genuinely new shrank every time it was checked more carefully. It started at about 7.8% of all proposals, dropped to around 3.5% once the catalog was expanded to include things like Python code libraries and command-line tools (a lot of "novel" ideas turned out to just be an existing code library with a different name), and dropped further, to somewhere between 1.5% and 4%, once a control test showed the judge itself called a tool "novel" about 22% of the time even when it clearly already existed. When a separate judge rated the truly novel proposals against ordinary ones blind, they came out equally good on every measure. Being "novel" didn't make an idea better. What did turn out to be useful: comparing what categories of tools models ask for against what categories actually exist in the real catalog. Models ask for search, science, and education tools about 6 times more often than the real world supplies them, while things like sports statistics APIs are built about 14 times more than anyone seems to want.
What we learned. Simply asking a range of AI models what tool they'd want is not a good way to discover genuinely new tools worth building. The vast majority of suggestions are things that already exist, and the sliver that looks novel keeps shrinking the more carefully you check it, largely because "novel" mostly reflects gaps in whatever catalog you're comparing against, not real gaps in the world. But the same setup is useful for something more modest: comparing what people, through their AI models, are asking for against what's already been built tells you which categories are underserved and which are oversupplied, without needing to judge whether any single idea is original.
Next week's competition
The thirty-seventh weekly competition is now open! Voting closes Friday, July 24 at 11:59 PM AOE.
Check out this week's ideas and upvote the ones that excite you. Submit your own ideas to enter the next round!
Last week we found that mapping how concepts sit inside a model doesn't predict what it actually does, and simply asking the model beats it; that fixing AI-generated training data's repetition problem with a base model breaks its connection to the source text without helping a model trained on it; and that asking AI models what tool they'd want mostly just returns things that already exist, though comparing that demand against what's actually been built is still a useful signal.
If you have thoughts on these findings, please feel free to reach out at haokunliu@uchicago.edu. We welcome collaborations and contributions! Check out our NeuriCo repo to see how the experiments are run.
If you are interested in citing this blog, use this bibtex:
@misc{liu-week-of-07-13-2026, author = {Liu, Haokun and Peng, Chenxi}, title = {Week of 07/13/26-07/19/26}, year = {2026}, month = {July}, day = {20}, url = {https://hypogenic.ai/blog/weekly-entry-260713} }