Autoresearch: an agent that runs the experiment loop

A generic workflow in which an LLM agent proposes, trains, evaluates, and accepts or rejects experiments on its own, inside rules a person sets. I built one instance of it for model compression; after manual validation of the first iterations, it ran a multi-day sequence of experiments without further input and delivered a materially smaller model within the performance tolerance.

2026 automated experimentation · agents · model compression
unattended
multi-day run after initial validation
performance-gated
every candidate accepted by rule, not judgment
reproducible
one versioned manifest per experiment

The pattern. Autoresearch is a simple idea that many people are converging on: hand the experiment loop to an LLM agent and keep the human on the rules. The loop is generic (propose a change, run it, read the results, decide, repeat), so the same scaffold applies to any problem where each candidate needs a training run, an evaluation, and a judgment call. A person can run a handful of these a day and loses context between them; an agent can run them around the clock and keep every result on the table.

The instance. I built one for model compression, where the search space is changes to features, architecture, and embeddings, and the acceptance criterion is a fixed performance tolerance.

What the workflow does. The agent owns the loop; the human owns the rules.

  • Search constraints and rules. I defined the space of allowed changes (feature, architecture, and embedding changes), how they may be alternated and combined, and what counts as an improvement. The agent proposes within that space.
  • Experiment manifests. Each candidate becomes a versioned manifest that fully specifies the run, so results are reproducible and comparable.
  • Concurrent training and inference jobs. The agent launches and tracks several runs at once and collects their artifacts.
  • Analysis of logs and evaluation results. It reads training logs and evaluation output, diagnoses failed or degenerate runs, and summarizes what moved.
  • Performance-gated acceptance. A candidate is accepted only if performance stays within a fixed tolerance of the baseline. The gate is a rule, not a judgment call, so the agent cannot talk itself into a bad trade.
  • Combine, then retest. When several candidates pass the gate in the same round, the agent combines them and runs the combination. If the combination also passes, it becomes the new baseline. If it does not, the agent keeps the single change with the largest compression gain and continues from there.
Human: rules search space · constraints performance gate Propose next change → manifest Run concurrent train + inference jobs Analyze logs · eval results · diagnosis Performance gate accept if Δperformance within tolerance Combine survivors, retest else keep the single biggest win new baseline, next round
The human sets the rules once; the agent runs propose → run → analyze → gate → combine for as long as the budget allows.

What happened. After I validated the first few iterations by hand, the workflow ran on its own for days, chaining dozens of experiments. The accepted sequence of changes produced a materially smaller model while staying within the performance tolerance.

What I learned. The agent is only as good as the rules it is handed: a precise search space and a hard acceptance gate mattered more than any prompt. Reproducible manifests made the run auditable after the fact, which is what let me trust the result. The scaffold itself is problem-agnostic; swapping the search space and the gate is enough to point it at a different question.

The other lesson is that none of this needed a framework. I started out assuming I would have to adopt someone else’s autoresearch codebase. Instead I built the loop together with a coding agent, and after a few rounds of that I understood exactly what the loop needed and where to hand it over. Autoresearch is something anyone can build for their own problem; the idea is simple, and the work is in the rules.