towards data science
towards data science on Beyond Market Intelligence: a running collection of 125 stories we have gathered and hand-picked because they are worth your time. Every post here touches on towards data science in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around towards data science, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.
“Los Movimientos”: The Routing Problem That Nearly Broke My Spirit
Facing a complex pickup-and-delivery problem with tight time windows? “Los Movimientos”: The Routing Problem That Nearly Broke My Spirit details a challenging optimization journey, demonstrating how mathematical techniques can tackle real-world logistical hurdles. This post explores the intricacies of routing, offering practical insights for anyone grappling with similar constraints. Discover how careful problem formulation and optimization algorithms can yield surprisingly effective solutions—a process that underscores the power of data science.
How I Reproduced BM25, Dense Retrieval, and SPLADE on a 16GB MacBook
Reproducing retrieval baselines—BM25, Dense Retrieval, and SPLADE—on limited hardware presents unique challenges. This practical exploration details the process of implementing these techniques on a 16GB MacBook, outlining the inevitable crashes, critical fixes, and essential score checks vital for building robust Retrieval-Augmented Generation (RAG) systems. Gain insights into real-world implementation hurdles and solutions. For further exploration of optimizing data workflows, consider "Reducing Human Annotation with ML Active Learning."
Reducing Human Annotation with ML Active Learning
In today's data landscape, human annotation represents a significant and often overlooked expense. Discover how Machine Learning Active Learning can transform this process, ensuring your team focuses their expertise only where it’s truly needed. This approach intelligently prioritizes data points requiring human review, maximizing efficiency and accelerating model development. Explore the power of targeted annotation—it’s a future-focused strategy for streamlining workflows and optimizing resources. For a deeper dive into related optimization challenges, see "Los Movimientos," which details tackling complex routing problems.
How to Give an LLM Agent a Browser
Empower your LLM agents to navigate the web with confidence. This guide explores building a browser-enabled agent using OpenAI's Agents SDK and Playwright’s MCP, unlocking a new dimension of data access and automation. Discover how to equip your AI with the ability to interact with websites, extract information, and perform tasks previously beyond its reach. This approach moves beyond static datasets, enabling dynamic, real-time data processing. For further insights into AI agent capabilities, see "You Can Hand One AI Agent Your Worst Recurring Task.
Build and Run an Intelligent Document Processing (IDP) System in the Cloud
Unlock streamlined data management with an Intelligent Document Processing (IDP) system, now accessible in the cloud. This guide details building and running a solution on AWS to automate the classification and extraction of Personally Identifiable Information (PII) from emails – a critical step for compliance and efficiency. Discover how to transform unstructured data into actionable insights, empowering your workflows. For a deeper dive into the foundation models underpinning such systems, explore "Tabular LLMs: An Introduction" on our site.
Loop Engineering for RAG Generation: An LLM Cascade from a Cheap Local Model Up to a Hosted Flagship
Loop Engineering presents a compelling approach to Retrieval-Augmented Generation (RAG) with its LLM Cascade, detailed in "Loop Engineering for RAG Generation." This innovative strategy sequences language models, starting with cost-effective local models and scaling up to a hosted flagship, optimizing both expense and accuracy. The research validates this cascade through rigorous testing—a sweep of twenty local models compared against a flagship—highlighting two key benefits: cost efficiency and a robust validation loop.
Tabular LLMs: An Introduction to the Foundation Models That Predict Your Spreadsheet
Tabular foundation models represent a significant shift in data management. These innovative models predict missing spreadsheet columns zero-shot—akin to how large language models complete text—and are rapidly surpassing traditional gradient-boosted trees on benchmarks like TabArena. Our exploration details how these models function, features an independent reproduction of a leading open-source implementation, and clarifies where XGBoost maintains its edge. For a deeper dive into AI assistants, consider exploring "Bluesky’s AI assistant Attie expands into an open social research tool."
Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract
RAG systems, while promising, frequently produce inaccurate outputs. Our latest research, "Most RAG Hallucinations Are Extraction Errors," reframes this issue, clarifying that many perceived “hallucinations” stem from flawed data extraction, not imaginative generation. We identify seven typed-contract patterns designed to ensure generation honesty, particularly valuable for smaller models. This decomposition rule significantly improves reliability. Addressing the AI context gap—as explored in our article on enterprise AI trust—is critical for realizing RAG’s potential.
Lessons Learned After 8.5 Years of ML
After 8.5 years immersed in machine learning, certain core principles consistently emerge. Patience is paramount; progress isn't always linear. Optimism fuels exploration, while discipline ensures rigorous execution. Successful ML isn’t solely about algorithms—it’s about well-defined projects and high-performing teams. These lessons underscore the importance of a grounded, iterative approach. For a deeper dive into practical challenges, consider "Most RAG Hallucinations Are Extraction Errors," which highlights critical error identification in retrieval-augmented generation systems.
Why Adding More AI Agents Made Our System Slower
Scaling AI agent systems isn’t always linear. We recently encountered a surprising bottleneck: asynchronous task management. As we expanded to hundreds of LLM agents, seemingly minor CPU tasks quietly became our largest performance constraint, slowing overall system speed. This post details how we identified and addressed this hidden cost, offering practical insights for anyone building complex AI workflows. Learn from our experience – a challenge we’ve explored further, alongside broader lessons from 8.5 years of machine learning.
Loop Engineering for RAG Generation: iterate top-k one at a time
Loop Engineering’s latest research explores a refined approach to Retrieval-Augmented Generation (RAG), detailing “iterate top-k one at a time” for enhanced Enterprise Document Intelligence. This methodology introduces two distinct regimes for candidate selection and a crucial "sufficiency signal" to optimize generation. Notably, per-question type dispatch significantly reduces processing costs. This innovative technique represents a crucial step towards more efficient and intelligent AI applications—a shift mirrored by companies like Monday.com, who are streamlining operations to focus on AI.
Build an LLM Agent That Can Write and Run Code
Unlock the potential of AI-powered code generation and execution. This hands-on walkthrough guides you through building an LLM agent using the OpenAI Agents SDK and Docker. Learn to empower your workflows by seamlessly integrating code writing and running capabilities. We’ll demonstrate a practical approach to leveraging these tools, offering a future-focused solution for data professionals. For those interested in a deeper dive into LLM runtimes, explore "How To Build Your Own LLM Runtime From Scratch" for a comprehensive understanding of the underlying infrastructure.
How To Build Your Own LLM Runtime From Scratch
Ever wondered what it takes to build an LLM inference runtime from the ground up? This comprehensive guide details that journey, walking you through the creation of a small runtime called annotated-llm-runtime, all while running on an H100. We explore the intricacies of managing weights and CUDA graphs, highlighting three key bugs that shaped the development process. Delve into the complexities of AI infrastructure—as explored further in "OpenAI’s AI spending spree has ballooned to $750B"—and empower yourself with a deeper understanding of LLM technology.
Loop Engineering with Adaptive Parsing in Action: Parsing Flat Tables with Azure and Figures with a Vision LLM
Loop Engineering presents a progressive approach to enterprise document intelligence, demonstrating Adaptive Parsing in action. This initial installment, "Parsing Flat Tables with Azure and Figures with a Vision LLM," explores utilizing Large Language Models (LLMs) as a critical last line of defense. We detail two complete escalations: extracting data from flat tables via Azure and interpreting figures through a vision model. For those seeking to optimize agent performance, consider "How to Run Claude Code Agents for 24+ Hours" for deeper insights into long-running coding agents.
How to Run Claude Code Agents for 24+ Hours
Unlock sustained coding productivity with Claude Code Agents running continuously – even for 24+ hours. This guide explores how to leverage these powerful AI assistants to streamline your engineering workflows and tackle complex projects with unprecedented efficiency. Discover practical techniques for maintaining and optimizing long-running agents, transforming your coding process. For a foundational understanding of setup and configuration, see "A Beginner’s Guide to Setting Up Claude Code for High Performance Agentic Programming" and elevate your agentic programming skills.
Automatically Assign a Category to Uncategorized Rows in Power Query and DAX
Categorized data is foundational for effective reporting and analysis; uncategorized rows hinder grouping and aggregation. When faced with data lacking assigned categories, establishing rules for assignment becomes essential. This post explores a practical solution for automatically assigning categories to uncategorized rows, demonstrated through a facility management project using Power Query and DAX. Discover how this approach unlocks deeper insights from your data. For further exploration of related techniques, see "TabFM Studio" and its application to spreadsheet predictions.
Loop Engineering for RAG Question Parsing: The Small Loop That Runs Before Retrieval
Optimizing Retrieval-Augmented Generation (RAG) systems hinges on precise question parsing. Loop Engineering for RAG, detailed in our latest Enterprise Document Intelligence report [Vol.1 #6quinquies], introduces a streamlined approach: a deliberately small loop focused on question refinement. This involves reading the document, identifying gaps, and re-parsing the query—a critical step before retrieval. Explore this technique to enhance accuracy and efficiency. For a foundational understanding of iterative learning processes, consider “Backpropagation Explained for Beginners (Part 1).”
Many Companies Use AI. Few Know How to Build an AI-Native Enterprise Data Platform.
Many companies are leveraging AI, yet few possess a practical architecture for an AI-native enterprise data platform. Building one demands more than isolated AI tools; it requires a cohesive system. Our latest article explores a robust architecture featuring data agents for streamlined integration, AI-powered quality assurance, and essential AI governance. Discover how to move beyond experimentation and establish a foundation for scalable, reliable AI initiatives. For related insights on structuring data for AI agents, see Pinecone’s introduction of Nexus Engine.
Loop Engineering with Adaptive PDF Parsing: Start Cheap, Pay for a Heavier Parser Only When the Page Needs It
Loop Engineering’s adaptive PDF parsing offers a transformative approach to document intelligence. Start with a cost-effective parser and only escalate to heavier processing when a page demands it—ensuring you pay only for what you need. This innovative system incorporates an escalation cascade and deterministic checks, proactively flagging parse failures *before* incurring deeper processing costs. Discover how this model delivers efficiency and predictability for enterprise document workflows, as explored in detail in our Enterprise Document Intelligence series.
Using Classical ML to Empower AI Agents
AI agents are rapidly evolving, but achieving true operational efficiency requires more than just the latest neural network architectures. A pragmatic approach involves leveraging the proven strengths of classical machine learning. This post explores the significant value of building upon existing ML foundations to empower AI agents, ensuring stability and predictable performance. We’ll examine how integrating established techniques can address key challenges in agent design.
Analog AI Is Back, But Can It Survive Its Own Noise?
The resurgence of analog AI presents a compelling solution to AI's escalating energy demands, leveraging physics rather than digital logic for computation. This exploration delves into how these chips function, revisiting a technology previously hampered by inherent noise. We examine the challenges that nearly sidelined analog computing and demonstrate the impact of simulated noise firsthand. For a broader perspective on AI deployment challenges, see "QCon AI Boston: Production AI Moves Beyond Prompts to Platforms, Harnesses, and Evals."
One RAG Pipeline, Four Very Different PDFs: Same Four Bricks, Every Answer Typed and Cited
Harnessing the power of Retrieval-Augmented Generation (RAG), our latest Enterprise Document Intelligence report, Vol. 1 #9B, demonstrates a single RAG pipeline effectively processing four diverse PDFs—a NIST standard, a report with a broken table of contents, and more—all while providing fully typed and cited answers. This approach underscores the transformative potential of AI-native document understanding. Explore how a unified architecture can bridge disparate data sources and deliver actionable insights. For deeper understanding of question parsing within RAG systems, see "Context Engineering for RAG Question Parsing."
Prepare These 5 Assets Before Your AI Agents Take On More Work
Ready to empower your AI agents to handle more work? Success hinges on thoughtful preparation. Before scaling AI adoption, prioritize defining recurring tasks, providing the right contextual data, and establishing clear benchmarks for high-quality output. Critically, determine where human judgment remains essential. These five assets are foundational. As Amazon’s AGI director recently highlighted, reliability—not just capability—is key to enterprise AI deployment; explore deeper insights on this challenge in "Amazon AGI director says AI agent reliability…”.
Why Your Betas Explode: The Hidden Geometry of Multicollinearity
Regression coefficients behaving unexpectedly? The phenomenon of “exploding” betas often stems from a less-discussed culprit: multicollinearity. This post unveils the hidden geometry behind this statistical challenge, explaining why highly correlated predictors destabilize your models. Discover how understanding the underlying geometric relationships – specifically, the angle between variables – can illuminate coefficient volatility and guide effective feature selection. Explore practical strategies to diagnose and mitigate multicollinearity, ensuring stable and reliable regression results.