If an AI evaluates an argument differently simply because of who it thinks said it, that's not a bug report, it's a mirror. The new preprint from chronosAndCode reveals something uncomfortable but essential: large language models are deeply sensitive to source attribution, and not in the way we might assume. The most consistent effect wasn't a reward for relevant expertise, but a penalty for irrelevant authority. When an argument was attributed to an expert in an unrelated field, several models judged it less credible than if no source was given at all. That pattern tells us more about how these systems mirror human cognitive shortcuts than we might like to admit.
This study deserves attention because it isolates a variable most of us treat as background noise. The researcher tested twenty arguments across ten topics with four source conditions, running them through GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek Flash. Credibility was the dimension that shifted most dramatically with source changes, while logic and agreement stayed relatively stable. That finding aligns with something we explored recently in How AI's Confidence Boost Can Mask a Real Drop in Developer Speed: the gap between what a system projects and what it delivers. Here, the projection comes from an attached name rather than a confidence score, but the risk is the same. We stop questioning the substance when the wrapper looks right.
The open question the author poses is the one worth holding onto. Should source sensitivity in an LLM evaluator be considered a bias when source expertise is genuinely relevant, or is the real problem specifically when the model overweights irrelevant authority? Our view is that the problem is the latter, and it is the harder one to solve. A model that correctly defers to a cardiologist on heart health and ignores a botanist is doing something useful. But a model that penalizes a solid argument because it was attributed to someone outside the expected field is replicating a very human failure: credentialism dressed as critical thinking. This is not the same as the automated model selection we saw in Cloudflare's AI Gateway Picks the Right Model for Each Request Automatically, where a classifier routes tasks based on capability. That is a technical optimization. This is a judgment call with no clean ground truth.
The practical takeaway for anyone building with or evaluating LLMs is straightforward: treat source attribution as a variable you must control, not a detail you can ignore. If you are using an AI to assess arguments, proposals, or customer feedback, the reputational weight of a name can skew the result more than the content itself. The preprint is not yet peer-reviewed, and the author invites criticism of the design and interpretation, that is the right posture. But the core observation is robust enough to act on. Watch for the irrelevant-expertise penalty in your own workflows. It is the shape of a bias that looks like sophistication but functions like a shortcut, and it will quietly shape the decisions your tools make.