Definity embeds agents inside Spark pipelines to catch failures before they reach agentic AI systems
Our take
In the fast-evolving landscape of data engineering, the ability to ensure pipeline reliability is becoming increasingly critical. Traditional methods, which often rely on post-failure alerts and manual troubleshooting, are insufficient for the demands of modern enterprises, especially as they integrate AI into their operations. The recent advancements from Definity, a Chicago-based startup, highlight a significant shift in how data pipeline management can be approached. By embedding agents directly within Spark or DBT drivers, Definity is poised to redefine the reliability of data pipelines, moving from reactive to proactive measures. This is particularly relevant as businesses increasingly depend on accurate and timely data to fuel their AI systems, where a single failure can have far-reaching consequences. As explored in other recent discussions, such as Job has me doing a needlessly complicated task and Build AI Financial Models in Sourcetable, the push for efficiency and simplicity in data processes is a common theme across the industry.
Definity's approach addresses a critical gap in the existing data pipeline monitoring landscape, which often operates from an outside observation point. Current tools like Datadog or Acceldata tend to react after an issue has occurred, leading to wasted compute resources and delayed responses. This delay can be detrimental in environments where data freshness is paramount. By installing a JVM agent directly within the pipeline execution layer, Definity enables real-time monitoring and intervention. This capability allows teams to not only address issues as they arise but also to prevent the propagation of bad data before it can impact downstream processes. The feedback from early users, such as Nexxen, who reported a 70% reduction in troubleshooting efforts, underscores the effectiveness of this model. It illustrates a transformational shift in how data engineering teams can operate—moving from a cycle of reaction to a state of continuous optimization.
What stands out in Definity’s model is its emphasis on providing full-stack context that is both real-time and production-aware. This goes beyond mere observation to enable teams to act decisively when issues arise. The ability to modify resource allocations or halt a job mid-execution based on current data conditions is a game-changer for enterprises that rely heavily on accurate data for decision-making. As data engineering increasingly intersects with AI workloads, the need for such proactive measures becomes even more pronounced. The implications for enterprise data teams are significant; they are no longer merely supporting analytics but are now integral to the delivery of AI-driven insights. This transition necessitates a reevaluation of how organizations perceive and manage their data infrastructure.
Looking ahead, the question remains: how will organizations adapt to these new capabilities? As teams begin to embrace in-execution intelligence, the potential for reduced operational costs and enhanced data reliability could lead to a fundamental shift in how data is utilized across industries. The urgency to adopt such innovative solutions could redefine competitive advantages in sectors heavily reliant on data-driven decision-making. As organizations continue to explore these advancements, the focus will likely shift toward not just adopting new tools, but also fostering a culture of proactive data management that prioritizes agility and responsiveness in an ever-changing digital landscape. The future of data engineering is not just about managing data; it's about mastering it in real-time.
For most data engineering teams, managing pipeline reliability often means waiting for an alert, manually tracing failures across distributed jobs and clusters, and fixing problems after they've already hit the business. Agentic AI needs the data to be there, clean and on time. A pipeline that fails silently or delivers stale data doesn't just break a dashboard — it breaks the AI system depending on it.
That gap is what Definity, a Chicago-based data pipeline operations startup, is building into: embedding agents directly inside the Spark or DBT driver to act during a pipeline run, not after it. One enterprise customer identified 33% of its optimization opportunities in the first week of deployment and cut troubleshooting and optimization effort by 70%, according to Definity. The company also claims customers are resolving complex Spark issues up to 10x faster.
"You need three big things for agentic data operations: full stack context that is real time and production aware. Control of the pipeline. And the ability to validate in a feedback loop. Without that, you can be outside looking in and read only," Roy Daniel, CEO and co-founder of Definity told VentureBeat in an exclusive interview.
The company on Wednesday announced that it has raised $12 million in Series A financing led by GreatPoint Ventures, with participation from Dynatrace and existing investors StageOne Ventures and Hyde Park Venture Partners.
Why existing pipeline monitoring breaks down at scale
Existing tools approach the problem from outside the execution layer — Datadog, which acquired data quality monitor Metaplane last year, Databricks system tables, and platforms like Unravel Data and Acceldata all read metrics after a job completes. Dynatrace has monitoring capabilities; it also participated in Definity's Series A.
The Definity approach is differentiated from other options in the way the solution is architected. According to Daniel, that means by the time a platform monitoring tool surfaces a problem, the pipeline has already run — and the failure, the wasted compute or the bad data is already downstream.
"It's always after the fact," Daniel said. "By the time you know something happened, it already happened."
How Definity's in-execution agents work
The core architectural difference is where the agent sits — inside the pipeline rather than watching from outside it.
Inline instrumentation. The Definity system installs a JVM agent directly inside the pipeline execution layer via a single line of code, running below the platform layer and pulling execution data directly from Spark.
Execution context during the run. The agent captures query execution behavior, memory pressure, data skew, shuffle patterns and infrastructure utilization as the pipeline runs. It also infers lineage between pipelines and tables dynamically — no predefined data catalog is required.
Intervention, not just observation. The agent can modify resource allocation mid-run, stop a job before bad data propagates or preempt a pipeline based on upstream data conditions. Daniel described one production deployment where the agent detected that an upstream job had been preempted and the input table it was supposed to write was stale — and stopped the downstream pipeline before it started, before bad data reached any dependent system.
What is and isn't real time. Detection and prevention are real time. Root cause analysis and optimization recommendations run on demand when an engineer queries the assistant, with full execution context already assembled.
Overhead and data residency. The agent adds approximately one second of compute on an hour-long run. Only metadata transmits externally; full on-premises deployment is available for environments where no metadata can leave the perimeter.
What in-execution intelligence looks like in a production environment
One early user of the Definity platform is Nexxen, an ad tech platform running large-scale Spark pipelines for mission-critical advertising workloads, running on-premises.
Dennis Meyer, Director of Data Engineering at Nexxen, told VentureBeat that the core problem he was facing was not pipeline failures but the accumulating cost of inefficiency in an environment with no elastic cloud capacity to absorb waste.
"The main challenge wasn't about pipelines breaking, but about managing an increasingly complex and large-scale environment," Meyer said. "Because we operate on-prem, we don't have the flexibility of instant elasticity, so inefficiencies have a direct cost impact."
Existing monitoring tools gave Nexxen partial visibility but not enough to act on systematically. "We had existing monitoring tools in place, but needed full-stack visibility to understand workload behavior holistically and to systematically prioritize optimizations," Meyer said.
Nexxen deployed Definity with no pipeline code changes. According to Meyer, the team identified 33% of its optimization opportunities within the first week, and engineering effort on troubleshooting and optimization dropped by 70%. The platform freed infrastructure capacity, allowing the team to support workload growth without additional hardware investment.
"The key shift was moving from reactive troubleshooting to proactive, continuous optimization," Meyer said. "At scale, the biggest gap often isn't tooling — it's actionable visibility."
What this means for enterprise data teams
For data engineering teams running production Spark environments, the shift from reactive monitoring to in-execution intelligence has architectural and organizational implications worth thinking through.
Pipeline ops is becoming an AI infrastructure problem. Data pipelines that previously supported analytics now carry AI workloads with direct business dependencies. Failures that were once an inconvenience are now blocking production AI delivery.
Troubleshooting time is a recoverable cost. According to Meyer, Nexxen cut engineering effort on troubleshooting and optimization by 70% after deploying Definity. For teams running lean, that time going back to the roadmap is the most direct near-term case for evaluating this category.
Read on the original site
Open the publisher's page for the full experience