ScientistTwo: what Google's autonomous AI scientist really shows, and what it doesn't

A new paper from Google Cloud AI Research describes an AI system that takes a research problem from a published paper, beats the paper's own result, and writes up the work. We read it closely: here is what it does, what the numbers mean, and what every researcher using AI should take from it.

On 17 September 2026, a team at Google Cloud AI Research, with a co-author from the University of Waterloo, posted "ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI" (arXiv:2609.19644, cs.AI). The authors are Jaehyun Nam, Jinsung Yoon, Yanzhou Pan, Yubo Wang, Rui Meng, Parthasarathy Ranganathan and Tomas Pfister.

The claim is ambitious. Given a research problem, ScientistTwo reproduces the current best method, comes up with ideas to beat it, writes and runs the code, tests its own ideas with ablations, drafts a conference paper, and defends that paper against simulated peer reviewers, all without a human in the loop. The paper is shared under a CC BY 4.0 licence, and the project has a public site at scientist-two.github.io.

How ScientistTwo works

ScientistTwo is a team of specialised AI agents. At every stage, one agent proposes something and a "critic" agent decides whether to accept it, send it back for refinement, or reject it. The pipeline has six stages.

  1. Seed ideas. One agent lists the limitations of the current state-of-the-art method and another checks that list. An idea generator then proposes many ideas, and a novelty checker ranks them.
  2. Test small, then big. A coding agent first reproduces the human baseline on a subset of the benchmark. Each idea is implemented and tested on that subset. Ideas that look good, or can be fixed with some engineering, move on to the full benchmark.
  3. Evolve the ideas. An "evolver" agent writes new ideas based on what happened in earlier experiments, and these are mixed with untested seed ideas so the system keeps exploring. A selector then picks the best result.
  4. Ablations. The system plans and runs ablation studies to check which parts of the method actually produce the gain.
  5. Draft and rebut. A drafting agent writes a paper in ICLR format. An AI reviewer called ScholarPeer scores it on a 1 to 10 scale. If the score is below 8, a rebuttal agent plans and runs extra experiments, and the paper is revised. This repeats up to a set budget.
  6. Meta-review. A meta-reviewer either accepts the work or asks for a deeper change, in which case the system builds a new hypothesis and keeps it only if it is strictly better.

An integrity audit sits on top: it checks that reported scores match the code's output, that the task's specification was followed, that every reference is real, and that the method described in the paper matches the code. Most agents run on Gemini 3.6 Flash, while the coding-heavy agents use Claude Code with Claude Opus 4.8.

The results, in numbers

The team tested ScientistTwo on 107 problems taken from papers accepted at NeurIPS 2025 (38), ICLR 2026 (5) and ICML 2026 as spotlight papers (64). For each problem, the "human state of the art" is the result reported in the original paper.

MeasureResult reported
Problems where it beat the human result86 of 107 (80.4%)
Average improvement over the human result25.2% (median 7.7%)
AI review score, ScholarPeer (1 to 10)7.5 on average; 91.9% judged acceptable
AI review score, Stanford Agentic Reviewer (held out)5.7 on average; 72.1% judged acceptable
Best earlier autonomous system (ScientistOne)3.8 ScholarPeer score; 0% acceptable on the held-out reviewer
Human experts' overall rating vs human papers (1 to 5)3.0, i.e. parity
Average cost and time per problemabout $3,765 and about 2.5 days

Two comparisons stand out. Against the actual human-written papers, ScientistTwo's drafts scored slightly higher on the ScholarPeer reviewer (7.5 against 6.2 to 6.9 for the human papers). And compared with seven earlier "AI scientist" systems, the jump is large: none of the earlier systems produced a paper the held-out reviewer would accept.

The rebuttal loop does a lot of the work. Before any rebuttal, drafts scored 5.2 on ScholarPeer with 46.9% judged acceptable. After two rounds of rebuttal experiments, that rose to 7.6 and 93.9%.

Reading the numbers carefully

The results are impressive, but a few details matter before anyone concludes that AI now writes better papers than people.

  • AI is grading AI. The main quality scores come from AI reviewers, not human program committees. ScholarPeer is also the reviewer the system uses to improve its own drafts, so its scores are, in the authors' words, in-distribution. The held-out Stanford reviewer is the fairer test, and there the picture is more modest: 5.7 average and 72.1% acceptable, roughly level with the human papers rather than above them.
  • Human experts saw parity, not superiority. Nine human experts rated 33 of the papers. On their own the papers scored well (about 4 out of 5 for most sections), but against the human-written originals they came out level overall (3.0), and slightly behind on the method section (2.9).
  • The average gain hides a long tail. The 25.2% average improvement sits next to a 7.7% median, which means a few very large wins pull the average up. On the small ICLR 2026 set, the average gain was 3.8%, lower than the earlier AutoSOTA system's 7.2% on the same set.
  • One in five problems was not solved. 21 of the 107 problems did not produce an improvement.
  • It is expensive. At about $3,800 and two and a half days per problem, the authors themselves note this may put it out of reach for academic labs and independent researchers.
  • No breakthroughs yet. The authors say plainly that it does not yet reach the level of spotlight or oral papers, which need what they call paradigm-shifting conceptual breakthroughs. The gains are local improvements to existing methods.

The most useful finding: AI invents references

The integrity audit is, for everyday researchers, the most practical part of the paper. Before the reference-checking agent was added, the audit found 19 fabricated or unverifiable references out of 1,840 across the generated papers. After adding a dedicated reference-refinement agent, that dropped to zero. Without an agent that checked the paper's description against its code, 11 of 49 papers described a method that did not match what the code actually did.

About 1% of references sounds small, but in academic publishing one invented citation is enough to sink a paper or a reputation. If a heavily engineered system from a major AI lab produced them before it was explicitly checked, everyone using a general AI tool to write should assume theirs will too, and check every reference against the actual source.

What this means for researchers

  • AI is becoming a capable research collaborator on well-defined problems, especially where there is a benchmark to optimise. It is not yet a source of new ideas that change a field.
  • Simulated peer review works as a feedback loop. The jump from 46.9% to 93.9% acceptable after rebuttal rounds shows that structured, critical AI feedback genuinely improves a draft.
  • Verification is not optional. Scores, specifications, references and method-code agreement all needed separate checks. Researchers using AI should build the same habits by hand.
  • Human judgement still sets the bar. Expert reviewers placed the work at parity with good human papers, which is valuable but is not the same as replacing the people who decide what matters.

How this relates to Soba Researcher

Soba Researcher shares one idea with ScientistTwo: a draft improves when critical reviewers attack it and the paper is rewritten to answer them. Soba Researcher sends every draft to a panel of 120 AI Review Agents, merges their reviews into one decision letter, rewrites the paper, checks each required change, and runs a second fix for anything left unsolved.

The two systems are built for different jobs. ScientistTwo runs real experiments on GPUs and costs thousands of dollars per problem. Soba Researcher does not run experiments: it drafts and critiques the written paper, from a title or from your own dataset, and is free to start. That also means its results, quotes and references must be checked, just as the ScientistTwo integrity audit shows for any AI writing system. Read our FAQ for what Soba Researcher can and cannot do.

Key takeaways

  • ScientistTwo, from Google Cloud AI Research, beat the published result on 86 of 107 recent ML problems, with a median gain of 7.7%.
  • Its papers scored about as well as accepted human papers on a held-out AI reviewer, and human experts rated them at parity overall.
  • Its own audit found fabricated references until a dedicated checker was added, a lesson for anyone writing with AI.
  • It costs about $3,800 and 2.5 days per problem, and does not yet produce breakthrough-level work.

Source: Nam, J., Yoon, J., Pan, Y., Wang, Y., Meng, R., Ranganathan, P. and Pfister, T. (2026). ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI. arXiv:2609.19644. https://arxiv.org/abs/2609.19644. Licensed CC BY 4.0. Figures quoted are as reported by the authors; this article is an independent summary and commentary, not affiliated with Google.