evals
Beyond Market Intelligence keeps evals in one place: 6 stories so far. The section currently leads with “Taming AI's Real-World Demands: Agents, Evals, and Rising Token Costs”, “Anthropic restricts internal AI tests from live internet to ensure control”, and “Evolve Your Recommendations: Real-World Insights on Adaptive Systems”. When AI hits production, code arrives faster than teams can review it, tokens start costing real money, and agents need permissions that actually hold. Anthropic turned off live internet access for its internal AI evaluations, a move that signals a careful, deliberate approach to safety. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work… The list below is every evals story on Beyond Market Intelligence, newest first.
Taming AI's Real-World Demands: Agents, Evals, and Rising Token Costs
When AI hits production, code arrives faster than teams can review it, tokens start costing real money, and agents need permissions that actually hold. Engineers at QCon AI New York 2026 (December 15-16) are sharing how they manage these constraints, no theory, just what works. For a deeper look at how teams are building practical guardrails, see our piece on Anthropic restricting internal AI tests from live internet.
Anthropic restricts internal AI tests from live internet to ensure control
Anthropic turned off live internet access for its internal AI evaluations, a move that signals a careful, deliberate approach to safety. It's a smart act of control, not a sign of weakness. When agents can call real-world tools, a decision-first model can catch risky actions before they happen. Our piece on why a decision-first model can rein in risky AI agent actions explores that very idea.
Evolve Your Recommendations: Real-World Insights on Adaptive Systems
Adaptive recommendation systems are rarely defeated by model architecture alone. Mallika Rao shows that real complexity emerges in production, where real-time feedback loops, retrieval freshness, and multi-stage orchestration must all bend to latency and cost constraints. This is practical insight, grounded in operational reality. For deeper context on how such systems are built, explore our piece on *Agent Harnesses* to see similar engineering discipline applied elsewhere. Rao's work is a reminder that evolution happens under pressure, not in isolation.
Why enterprises double down on AI automation after being burned
Enterprises that have watched an AI agent pass its evals and then fail in front of customers aren't retreating from automation, they're moving faster toward removing humans from deployment decisions. That's the counterintuitive core of the latest VB Pulse research. Confidence in automated evaluation is climbing, yet the share of companies burned by test-passing failures remains stubbornly near half. The lesson isn't that agents are unreliable; it's that a passing score marks the start of monitoring, not the end of it.
Smarter Prompts, Not Bigger Ones: Engineering Context That Works
Big context windows feel like the easy answer, but Baruch Sadogursky and Patrick Debois argue otherwise in their session, "The Right 300 Tokens Beat 100k Noisy Ones." They show why stuffed prompts drag coding agents down, then offer practical fixes like lazy-loaded skills and externalized memory banks. It is a grounded, useful take for engineering leaders wrestling with agentic workflows.
How AI leaders are redefining product requirements through evaluations.
Expedia's Xavi Amatriain argues that evals have replaced the PRD as the starting point for product design. He's right to push this forward. Encoding what an agent should do, before writing code, forces teams to confront risk early. His three-layer governance model and risk-calibrated toll gates offer a practical alternative to brittle guardrails. It's a clear-eyed take from someone who's seen the stakes up close.