April 23, 2026•1 min read•from Machine Learning

AI scientists produce results without reasoning scientifically [R]

Our take

Recent research involving 25,000 experiments has revealed alarming trends in AI scientists: they often generate results without adhering to scientific reasoning. The findings show that 68% of the time, AI disregards gathered evidence, while 71% of AI models fail to update their beliefs altogether. Even when confronted with contradictory data, only 26% of the time do they revise their hypotheses.

Researchers ran 25,000 AI scientist experiments and discovered something that need attention!!

AI scientists are producing results without doing science.

68% of times, the AI gathered evidence and then completely ignored it. 71% times the AI never updated its beliefs at all. Not once. Only 26% of the time did the AI revise a hypothesis when confronted with contradictory data.

A human scientist adapts. You approach a chemistry identification problem differently than you approach a simulation workflow. The AI doesn't. It runs the same undisciplined loop every time.

The researchers also showed the most popular proposed fix: better scaffolding do not work.

Everyone building AI research agents has focused on engineering better prompting frameworks, better tool routing, better agent architectures. ReAct, structured tool-calling, chain-of-thought, all of it.

alphaxiv

arxiv

submitted by /u/Okra3268
[link] [comments]

Read on the original site

Open the publisher's page for the full experience

View original article →

[D] Are there REAL success stories of autonomous AI dev agents working reliably in production?I’m having a serious debate with a colleague, and I want to settle this with actual evidence instead of opinions. The claim: That it’s possible today to run orchestrated AI developer agents (multiple agents, coordinated workflows) that can autonomously build and maintain software — under supervision of a senior AI/dev — without running into unfixable errors or constant breakdowns. I’m skeptical. He believes it’s already happening. So I’m looking for real-world examples, not theory: - Have you actually used autonomous dev agents in production? - What was the setup? (tools, stack, orchestration method) - What level of autonomy are we talking about? - What still breaks? - Did it scale beyond small experiments or toy projects? Especially interested in: - Multi-agent setups (not just Copilot-style assistance) - Systems that run for extended periods (not one-off demos) - Cases where human input is minimal but still controlled If you’ve seen this work (or fail), I’d really appreciate detailed insights. Trying to separate hype from reality here. submitted by /u/MegaMillyMansion [link] [comments]

AI scientists produce results without reasoning scientifically [R]

Related Articles