Your AI agent for running and tracking ML experiments, all in one place.

Introducing Thesis, an innovative agent-native workspace designed to streamline the process of running and tracking machine learning experiments.

April 16, 20263 min readMachine Learning

Your AI agent for running and tracking ML experiments, all in one place. — Thesis: an agent-native workspace for running and tracking ML experiments [P]

The friction in machine learning has never been the models. It's the glue: the scripts that pass data to a trainer, the metrics logged in one place while the dataset lives in another, the notebook cell you rerun and hope produces the same result. Thesis takes direct aim at that fragmentation, and the move is worth paying attention to. By putting an agent in the loop to inspect datasets, launch runs, and monitor metrics from a single interface, the team is attacking the real bottleneck: context switching, not compute.

What stands out is the implicit critique of the status quo. Most practitioners live in a patchwork of Jupyter, a terminal, and a tracking server, and every hop between those tools costs more than time. It breaks flow, and flow is where the actual thinking happens. Thesis collapses that distance. The agent isn't a gimmick here; it's the natural interface for the job. You ask, it inspects, it launches, it reports back. That's not replacing the researcher. It's removing the clerical overhead that makes up most of the day.

The question the team poses to the community is the right one: where does this save time, and where do you still need a notebook? Our take is that the notebook isn't going away, nor should it. There will always be a place for exploratory, cell-by-cell tinkering. But the value of a tool like Thesis is in making the loop between experiment and analysis feel continuous rather than bolted together. If the agent can handle the drudgery of monitoring and iteration, the researcher can spend more time deciding what to try next. That's not a small thing. That's the difference between babysitting a run and actually thinking about the problem.

The practical takeaway is straightforward: the next wave of ML tooling won't be a better dashboard or a faster tracker. It will be an environment where the machine does the tracking and the human does the deciding. Thesis is positioning itself in that gap, and the early demo suggests it understands the pain. We'd tell practitioners to test it on a real, annoying workflow, the one with the messy data and the flaky metrics, and see if the agent earns its keep. If it does, the notebook becomes a tool for exploration, not a crutch for organization. And that would be a genuine shift in how we work.

From Machine Learning

We built Thesis, a workspace for running and tracking ML experiments with an agent in the loop. It can inspect datasets, launch training runs, monitor metrics, and help iterate on experiments from a single interface.
We're aiming to make model development less fragmented by combining experiment orchestration, run tracking, and agent-driven analysis in one place.

Read the original at Machine Learning

Briefing

Source: Machine Learning
Published: April 16, 2026

An autonomous AI researcher that refines itself on your data
Inspired by Andrej Karpathy's AutoResearch, I built a system where Claude Code acts as an autonomous ML researcher on tabular binary classification tasks (churn, conversion, etc.). You give it a dataset. It loops forever: analyze data, form hypothesis, edit code, run experiment, evaluate with expanding time windows (train on past, predict future - no leakage), keep or revert via git. It edits only 3 files - feature engineering, model hyperparams, and analysis code. Everything else is locked down. Edit: To clarify based on some comments, I am using this to solve the problem of finding new signals to add to the model, not trying to overfit a limited dataset. -end Edit- Key design decisions: Introducing an analysis loop in addition to the experiment loop, this allow for better reflection and experimentation. Optimize for experiment throughput with a bunch of decisions: Use LightGBM as default model, limit feature count and tree count, locking down training run until it finishes. Constrained editing surface: only 3 files + logs. No infrastructure changes, no package installs. Without this, the agent will eventually try to modify the evaluation code to "improve" its score. Docker sandbox - the agent runs with full shell access (--dangerously-skip-permissions). Container keeps it contained. Expanding time windows over k-fold - mean score across multiple temporal train/test splits. Forced logging - every experiment gets a LOG.md entry (hypothesis, result, takeaway). Significant insights go to LEARNING.md. You can read the agent's reasoning after the fact. Analysis primitives built-in - univariate AUC, correlation pairs, null rates, feature importance, error analysis. The agent writes analysis code using these to save time, they also serve as initial suggestions for the first few analyses. What I learned building this: Air-tight evaluation is the essential for real improvement - this lesson hit me twice: Earlier version didn't constraint which file the agent could edit, it eventually changed the evaluation code to make "improvement" easier for itself. K-fold validation was originally employed, the agent found improvements that are actually data leakage and didn't hold out-of-time. After a painful manual inspection, I switched over to expanding time windows. Do everything to protect experiment throughput - this lesson also hit twice: Initially, I let the model run wild and was not very impressed when it barely run 20 experiments overnight. Turns out, the agent engineered thousands of new features that slowed down training and crash some runs due to RAM limit. I added the feature count limit and tree count limit to make sure training time is reasonable. Despite that, the agent still manage to crash/slow down training runs by putting many of them into background process at the same time. -> Locking mechanism was implemented to prevent 2 experiments being run at the same time. After this, the rate of progress increased to hundreds of runs per day. Persistent memory is important: Without forced logging, the agent would repeat experiments it already tried. The LOG.md and LEARNING.md system gives it memory across iterations. The code open source (sanitized version): https://github.com/trantrikien239/autoresearch-tabularOf course it is done with Claude Code, but it has improved so much after rounds of iterations, including manual edits, so I think it's worth sharing. submitted by /u/Pancake502 [link] [comments]