AI-generated randomness is an oxymoron waiting to be exposed. A detailed analysis of open models, both base and instruction-tuned, reveals that the numbers these systems produce when asked to "pick a random number" follow a statistically predictable distribution. By reading the logits, the raw probability scores behind every token, a researcher could determine exactly how many guesses it takes to crack what users assume is a fair, unpredictable choice. The results are sobering: what feels like chance is actually a pattern waiting to be exploited.
This matters far beyond a single experiment. If your workflow relies on an AI to generate random numbers for simulations, lottery-style selections, or data sampling, you are trusting a system that was never designed to be truly stochastic. The same probabilistic engine that excels at predicting the next word in a sentence also leaks its internal preferences when asked for a "random" integer. We have seen similar patterns in how data science approaches ambiguity: From 719 Math Papers: Data Science Gets Smarter at Matching Messy Data shows how researchers are learning to handle imprecise text with structure and heuristics rather than raw randomness. The lesson here is parallel, when you assume unpredictability, verify the underlying mechanism.
The practical takeaway is straightforward: do not let an AI script your random-number generator for anything security-sensitive or scientifically rigorous. The logit analysis demonstrates that even instruction-tuned models, which are trained to follow user prompts more faithfully, still cluster their answers around predictable values. This is not a flaw unique to one model family; it is a property of how language models encode probabilities. In Seven Decision Models Put to the Test Against a Generative LLM, seven different decision frameworks were compared head-to-head, and the results showed that accuracy varies significantly depending on the task structure. Random-number generation is no different, treat it as a decision task with measurable bias, not a black box of entropy.
What we are left with is an open question every data practitioner should consider. If a supposedly random number from an AI is predictable within a handful of guesses, how many other "creative" outputs inherit hidden constraints from the model's training distribution? The next time you ask an AI to choose a starting seed, shuffle a list, or pick a winner, remember that the design goal was language fluency, not cryptographic fairness. That distinction is the difference between a useful tool and a compromised process.