benchmarking

Safety critical systems (SCS) are the only real benchmark for ML systems. Thoughts? [D]

Safety-critical systems (SCS)—like flight controllers, braking systems for high-speed trains, or reactor protection systems—represent the ultimate benchmark for machine learning’s real-world viability. Successfully deploying LLMs and neural networks within these demanding environments would not only sway skeptics but also address critical issues plaguing the field: the disconnect between benchmark performance and practical application, and the prevalence of overhyped claims. Demonstrating reliability in SCS would be a definitive test, moving beyond simulations and proving the transformative potential of AI.

We’ve got a workshop on production retrieval-augmented generation with open models, benchmarked end to end, thought it’d be relevant here [D]

Unlock production-ready Retrieval-Augmented Generation (RAG) with our upcoming workshop on August 29th. Led by AI Consultant Ben Auffarth, this hands-on session builds and benchmarks end-to-end RAG pipelines using entirely open models—no API calls required. You'll discover hybrid retrieval techniques, crucial reranking strategies, and robust evaluation using RAGAS. Explore cost and performance benchmarking for open-model deployments, all while incorporating guardrails from the outset. Learn more and register here: [https://www.eventbrite.co.uk/e/the-genai-build-lab-build-production-ready-rag-

We built the Agentic World Cup - LLMs that compete in 1v1 Soccer. [P]

Introducing the Agentic World Cup, a pioneering platform designed to bridge the “embodiment gap” in AI. We’re challenging Large Language Models to compete in 1v1 soccer, creating a unique training and testing ground for true embodied intelligence. Simply sign in, select your LLM, coach it with prompting, and submit it to compete. Final rankings will be published this Friday. This initiative also addresses a critical need for embodied benchmarking, as explored in our recent article, "Producing the World’s Cheapest Tokens."

AI startup Hark unveils first product: an affordable, fast computer use agent Hark Handoff

Hark, a new AI startup founded by serial entrepreneur Brett Adcock, introduces Handoff, a computer use agent (CUA) poised to transform how we interact with the open web. Achieving a leading 97.7 score on the Online-Mind2Web benchmark—outperforming models like GPT-5.4 and Claude Opus 4.8—Handoff offers autonomous task completion, from online ordering to candidate outreach. With significantly lower operational costs, Hark empowers users to explore a future where AI handles routine digital tasks. Sign-ups are open now at hark.

[R] CausalVLBench: Benchmarking Visual Causal Reasoning in Large VLMs.

How Much Does a Local LLM Actually Cost to Run? I Measured Every Watt on Apple Silicon

Curious about the true cost of running a local Large Language Model (LLM)? We measured it—every watt—on Apple Silicon, analyzing five models during sustained generation. This deep dive reveals real-world energy consumption at a $0.31/kWh rate, uncovering surprising results that align with RTX-3090 predictions, only amplified. Discover how your hardware choices impact operational expenses and explore the evolving landscape of AI compute. For context on broader industry trends, see “Recursive Superintelligence signs $410M compute deal with Amazon.”

Evaluated 6 frontier LLMs (GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.7, Gemini Pro/Flash, Grok 4.3) on political, gender, and racial bias across 8 benchmarks (~20,600 examples) [R]

A recent solo evaluation project rigorously assessed six frontier LLMs—GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.7, Gemini Pro/Flash, and Grok 4.3—across eight established bias benchmarks, encompassing over 20,600 examples. Findings reveal a consistent leftward political leaning among all models except Grok, despite its self-reported right-leaning stance. Notably, GPT-5.4 exhibited the highest refusal rate (20.3%) when addressing race-related inquiries requiring explicit racial identification. For deeper insights into AI memory systems, explore "Context Windows Forget What Matters." Full data and

Tri-Net v2: Open-source implementation of our Scientific Reports paper on unified skin lesion and symptom-based monkeypox detection [R]

We’re pleased to announce the open-source release of Tri-Net v2, the fully reproducible research framework detailed in our recent *Scientific Reports* (Nature Portfolio) paper on unified monkeypox detection. This implementation prioritizes transparency and accessibility, featuring a leakage-free data pipeline, multiple CNN backbones (including ConvNeXt-Tiny), ensemble strategies, and Grad-CAM explainability. Installation is streamlined via `pip install mpox-trinet`, and comprehensive documentation is available on GitHub.

Exploring continual learning without replay buffers: Our findings using dynamic task-similarity routing [P]

Continual learning often relies on memory-intensive replay buffers, but our research introduces Coincidex, an open-source framework that bypasses this limitation. By employing a dynamic task-similarity routing layer, Coincidex intelligently directs data streams without caching historical samples. Initial benchmarks demonstrate graceful transfer in clean task boundaries, achieving strong results without manual tuning. While performance can degrade with chaotic, long-tail task sequences, we share this lightweight alternative to empower exploration and welcome community feedback on addressing these challenges. Explore the code and architecture here: [https://github.com/rakib-nyc/coincidex](https://github.com/rakib-nyc