data quality

Avoiding Entity Key Drift in a Data Lake: Step 2, When Fuzzy Matching Stops Working

Data lakes often suffer from entity key drift, a challenge that normalization alone can’t fully resolve. Our latest post, “Avoiding Entity Key Drift in a Data Lake: Step 2,” details a critical juncture where fuzzy matching proves insufficient for reliable data cleanup. We initially developed a matcher to address this, but real-world testing revealed inherent limitations. This article outlines the resulting architecture, born from setting aside the matcher and charting a new course.

What We Miss About Missing Values

Missing values are a ubiquitous challenge in data science, yet their implications often go unexamined. "What We Miss About Missing Values" explores the hidden assumptions embedded within the data we *do* observe—recognizing that what's absent can be just as informative as what's present. This post delves into the biases introduced by missingness and offers a framework for more thoughtful analysis. For a related perspective on navigating complexity in data systems, see "Why RAG Complexity Should Be Earned."

How to assess if there is a strong signal in your dirty data [Project]

Uncertain about the strength of signal within your high-dimensional, real-world data? Introducing Entropic Scree, a new diagnostic tool for tabular datasets. It estimates key properties – signal volume, signal-to-noise ratio (SNR), intrinsic rank, decoupled sub-networks, and linear sufficiency – moving beyond traditional PCA limitations by evaluating transformed mutual information. This method offers broader applicability and serves as a practical diagnostic for the "From Garbage to Gold" framework. Explore the preprint and access the R function now; Python and R packages are forthcoming.

5 Python Libraries That Make Data Cleaning More Enjoyable

Data cleaning doesn’t have to be a chore. This article introduces five Python libraries designed to transform tedious data preparation into an expressive and genuinely enjoyable process. We've compiled a list of tools that empower you to streamline workflows and unlock deeper insights from your data. Discover how these libraries can simplify complex tasks and accelerate your analysis. For those working with image classification, you might find our accompanying dataset, "Starfield Fauna," a valuable resource for practical application.

An eval harness found what qualitative review couldn't: AI models are most confident when wrong

Many teams developing large language model (LLM)-assisted tools overlook a critical step: verifying the accuracy of model outputs against ground truth. While qualitative reviews assess fluency and coherence, they often miss confidently incorrect explanations – a significant risk when these tools inform real business decisions. A new evaluation harness reveals that AI models are surprisingly confident when wrong, highlighting the need for rigorous accuracy testing, particularly when building tools like root-cause explainers, as explored further in "I compiled Doom's renderer into a 21B-parameter transformer."

I Thought Loading Data Was the Finish Line. It Was the Starting Point.

Many believe data loading marks the end of a project, but it’s often just the beginning. My recent journey building dbt models illuminated the true meaning of "analysis-ready" data—a concept far beyond simply moving data from point A to point B. Discovering this shift transformed my approach to data management, emphasizing the importance of structured, reliable datasets. If you’re exploring the nuances of data transformation, consider "Before Q, K, and V: Reconstructing the Transformer" for a deeper look at foundational architecture.

Building Trustworthy Snowflake AI Agents with Semantic Governance

AI Slop Is Costing You Hours. Here's How To Stop Sending It.

AI-generated data errors – often called "AI slop" – are silently eroding productivity, costing teams countless hours in correction and rework. It’s a common problem, but not an inevitable one. Explore practical strategies to identify and mitigate these errors, reclaiming valuable time and ensuring data integrity. Discover how to refine your AI prompts and validation processes for more reliable outputs. For deeper insights into leveraging AI effectively, see our article, "Top 5 Claude Skills for Writing (Ranked by GitHub Stars)."

The Medallion Data Architecture: An Introduction

Navigating modern data pipelines can feel complex, but the Medallion Data Architecture offers a clear, practical framework. This guide introduces the Bronze, Silver, and Gold layers—a proven approach to structuring data for reliability and analytical readiness. We’ll explore each tier with a working Python and DuckDB example, empowering you to build robust data workflows. For a deeper dive into related challenges in AI agent memory management, see "Asana's AI agents share memory across your company — but not your secrets."

Do Legacy Organizations/Government Have More AI Talent Than AI Problems?

Many organizations, particularly legacy institutions and government entities, possess significant AI talent but face a surprising bottleneck: a lack of foundational data maturity. Discussions often leap to advanced AI solutions like RAG and agent frameworks before addressing core issues—data accuracy, governance, and accessibility. Before pursuing autonomous agents, establishing reliable data pipelines and answering fundamental questions about data origins and ownership is critical. As explored in "Stop Graphing Everything," even seemingly advanced techniques benefit from a solid data foundation.

Companies are finally seeing AI ROI — and now they know how much more value it can deliver

Companies are finally realizing the substantial ROI of AI, and the SAP Value of AI Report 2026 reveals just how much further that potential extends. Based on a survey of over 2,600 business leaders, the report indicates AI now supports nearly one-third of organizational tasks, with ROI expectations significantly increasing. However, realizing this full potential hinges on strategic data governance—a challenge many organizations are only beginning to address. Explore the full findings and discover how to unlock transformative value with AI.

Avoiding Entity Key Drift in a Data Lake: Step 1, Normalization

High-frequency streaming data pipelines against live APIs expose critical challenges, particularly entity key drift in data lakes. This four-part series tackles that challenge, beginning with normalization, a foundational step for maintaining data integrity. We'll explore building a robust pipeline using openSenseMap—a citizen-science IoT network—demonstrating how real-world data quality issues demand innovative solutions. Subsequent articles will cover matching, polling, noise filtering, and a vendor-agnostic Iceberg pipeline. Interested in broader data science considerations? See "What Professionals Should Know About Data Science and AI" for vital insights.

What Professionals Should Know About Data Science and AI, According to Harvard Business School Online

## What Professionals Should Know About Data Science and AI, According to Harvard Business School Online Harvard Business School Online highlights a critical truth: successful data science and AI initiatives hinge on fundamentals, not just the latest technology. Prioritize clear business goals, rigorous data quality, and simple, well-validated models. Realistic cost assessments and incorporating human judgment are equally vital. Don't chase complexity; instead, build a solid foundation.

AI agents aren't confidently wrong because of bad context — they're wrong because of bad data engineering

AI applications are increasingly delivering confidently incorrect answers, not due to model flaws, but a critical gap in data engineering. These failures occur when outdated or incomplete data is retrieved and presented as authoritative, bypassing standard data pipeline checks. Addressing this requires a shift in focus—from pipeline completion to data correctness, freshness, consistency, and lineage. Prioritizing these four dimensions of data observability is the key to building truly trustworthy AI systems.

The AI Slop Problem Nobody's Talking About | Substack CEO Interview

The current excitement around AI agents often overlooks a critical challenge: the "AI Slop Problem." Substack CEO Chris Best recently shared his insights on this phenomenon – the tendency for AI outputs to be messy, inconsistent, and difficult to manage. This interview dives into the core issue and potential solutions for building reliable AI systems. For a deeper exploration of architectural approaches moving beyond rudimentary AI, see our piece, "Presentation: From Copy-Paste to Composition." It’s time to address the unseen complexities hindering AI’s true potential.