Here's a curious thing about language models: we assume that words with many meanings produce messy embeddings. More dictionary senses ought to mean more geometric fragmentation. That intuition is clean, intuitive, and wrong.
A Reddit researcher working on word-sense disambiguation at home tested the assumption directly. They measured mean pairwise cosine similarity across 192 words using Qwen2.5-7B, extracting embeddings at layer 10. The correlation between WordNet sense count and embedding variance was essentially zero, Spearman rho of -0.057, p-value 0.43. What did predict variance was word frequency: rho of -0.239, p-value 0.0008, a result that held even after controlling for polysemy. The word "break" has sixty WordNet senses, but the model treats them as variations on a single theme. Meanwhile "face" has fewer formal senses yet gets pulled in multiple directions by its diverse co-occurrence patterns. Frequency, not semantic complexity, is the driver of embedding dispersion.
This matters for anyone building retrieval systems or fine-tuning embeddings. If high-frequency tokens are geometrically promiscuous because they appear everywhere, not because they mean many things, then they become unreliable query terms in RAG pipelines. A word like "get" or "set" may anchor a retrieval step less effectively than a rarer but semantically precise term. The researcher proposes a Contextual Promiscuity Index, a per-word, per-model, per-domain score for how dispersed a word's embeddings are across contexts. That index could flag noisy tokens during pretraining, guide precision allocation in embedding table compression, or identify which terms to handle with special attention in retrieval.
The retrieval experiments the researcher ran pointed in the right direction but lacked statistical significance, the corpus was roughly a thousand documents, and compute was limited. That is a call for replication, not dismissal. The finding itself is clean, falsifiable, and immediately useful: frequency shapes embedding geometry more than dictionary definitions do. If you manage a retrieval pipeline or compress embedding tables, start checking whether your high-frequency tokens are pulling their weight. The answer might be that they are just showing up everywhere, meaning nothing in particular.