Using Jensen-Shannon Divergence to detect narrative regime shifts in daily news corpora [P]

Most people building news scoring systems start with sentiment. It feels intuitive, and the tooling is abundant. But as one practitioner recently demonstrated, sentiment turns out to be the least interesting layer of the problem. The real signal lives in narrative regime shifts — the moments when the story a market is telling itself quietly changes direction before any aggregate score catches up. This work uses Jensen-Shannon Divergence across two dimensions: unigram and bigram frequency distributions, and a distribution of eight narrative frames ranging from Growth Momentum to Regulatory Risk. It is a method worth taking seriously, because it reframes what we think a spreadsheet or a data pipeline should actually be doing for us.

The core insight is deceptively simple. A corpus can maintain positive aggregate sentiment while the underlying framing drifts toward regulatory and geopolitical concern. That drift is visible in the frame distribution long before it registers in any sentiment metric. Two readers tackling complex data problems this week highlighted why this matters in practice. One was simplifying a task assignment process where 2000 tasks are broken up among 10 workers, recognizing that the real bottleneck was not volume but the inability to see structure in the data. Another was trying to get a bar graph to only show Yes percentages, discovering that filtering alone does not solve the problem when the display layer obscures what the numbers are actually saying. Both situations share a common root: the tools were tracking the obvious metric while the meaningful shift was happening somewhere else.

What makes the JSD approach compelling is that it treats narrative as a distributional problem rather than a keyword problem. You are not searching for specific phrases. You are measuring how the shape of the conversation changes over time. The stop-word list that strips AI and finance boilerplate is a small detail that carries outsized importance — without it, the signal drowns in institutional language that looks stable even when the underlying sentiment is shifting. The eight-frame taxonomy is where the author expresses the most uncertainty, and rightly so. Eight categories may be collapsing distinctions that matter, and the seven-day rolling window introduces noise that forces empirical threshold calibration from roughly 60 days of production data. That fragility is real, and it points to a broader challenge in applying distributional distance measures to short-horizon, high-frequency corpora. The literature on JSD tends to operate over much longer timescales, and there is a genuine gap in guidance for daily granularity.

The question worth watching is whether this kind of narrative detection becomes something practitioners can do with accessible tooling, or whether it remains a research-adjacent technique that requires specialized calibration. The methodology is published, the logic is sound, and the intuition — that reframing precedes scoring — feels durable. What remains open is whether the window sizing and taxonomy decisions can be made more robust, or whether this is one of those cases where the operational reality of daily data simply demands more iterative refinement than the theory accounts for.