AutoAnalysis puts human–AI teamwork against a tougher baseline

An AI assistant can improve someone’s work without making the resulting team better than its strongest member. That distinction changes the question an organization should ask when it tests a new workflow: better than a person working alone, or better than either a person or the AI?

AutoAnalysis, a preprint submitted on October 30 by Michelle Vaccaro, Abdullah Almaatouq and Thomas W. Malone, brings that question to a growing evidence base. Its October 26 data release combines 1,955 effect sizes from 420 papers. The authors report that human–AI combinations outperformed humans alone on average.

The tougher comparison was available for only 171 effect sizes, where experiments also measured AI working alone. On that subset, the combined systems underperformed whichever standalone partner did better. The result concerns an average across those experiments, not every team or task.

Two different tests

The distinction predates the new system. The authors’ peer-reviewed 2024 Nature Human Behaviour review examined 106 experiments, requiring measurements of humans alone, AI alone and the combination. It called improvement over humans alone augmentation, and improvement over both standalone alternatives synergy. Those are different performance tests; a positive result on the first does not establish the second.

Research design matters, too. An October 15 Max Planck Institute for Human Development account describes a separate reanalysis of the older review’s 74 papers. Teams tended to outperform the better partner when people received both explanations and feedback about correct outcomes. But none of those papers experimentally varied learning opportunities: the association does not prove that adding feedback caused the improvement, or that it transfers to today’s generative AI.

A review that keeps moving

AutoAnalysis uses AI agents to search, screen, extract and synthesize evidence on a recurring basis. The preprint itself has not been peer-reviewed, although its inclusion rules require English-language, peer-reviewed experiments.

Its pooled results vary substantially across studies. They do not establish long-term effects on learning or organizational performance. The authors also acknowledge that agents can miss papers or extract statistics incorrectly.

The authors checked 30 papers at each of three pipeline stages. They describe those small, internal audits as a sanity check, rather than a formal reliability study, and call for larger independent audits. Updating the evidence automatically still leaves the accuracy of that update to be demonstrated.