AI

We Run on AI Agents. Here's What Still Needs a Human.

Agents can now take on hours of work, but they grade themselves too kindly and fail most real-world briefs. What the 2025–26 research says the human is for.

This network runs differently from most publications. AI agents do most of the work behind each post: the research, the drafts, the fact-checking passes, the brand files. People set the direction and make the final calls.

So before building any further, we needed an answer for ourselves. What can agents now do reliably, and where do people actually matter? We went through the evidence, mostly from 2025–2026: independent agent tests, newsroom experiments that worked and ones that blew up, platform rules and audience surveys.

Agents can now take on hours of work at a time, but they can’t reliably tell you when they’ve got it wrong. The human job has moved from doing the work to briefing it, checking it and owning it. That’s good news for anyone building with them, because each of those jobs can be learned.

Agents can do long tasks, just not reliably

The research group METR measures how long a task an AI agent can complete, timed by how long the same task takes a skilled human. Its May 2026 report looked at the best publicly available agents on software, machine-learning and cybersecurity tasks. They could finish tasks that take an expert about 12 hours, but only half the time. At an 80% success rate, the task length drops to about 1.5 hours.

METR’s original study, peer-reviewed at NeurIPS 2025, found the same gap across models: the task length at 80% success is 4 to 6 times shorter than at 50%. The 50% figure has doubled roughly every seven months since 2019, and that pace may have sped up in 2024.

For anything published under your name, a coin flip isn’t good enough. So we plan to the 80% number. Big jobs get broken into pieces an agent can do reliably, with a check between each piece. As agents improve, the pieces can get bigger.

Precise briefs succeed. Real projects mostly don’t

OpenAI’s GDPval benchmark asks expert graders to compare a model’s work with a professional’s. OpenAI reports that its GPT-5.5 model’s work was rated as good as or better than the expert’s on 84.9% of tasks. That is OpenAI reporting its own model’s score on a benchmark OpenAI built. The benchmark’s own paper describes its tasks as “precisely-specified and one-shot”, with all the context supplied up front.

Real work looks different. The Remote Labor Index gives agents 230 real freelance projects. The best agent’s work is now accepted on 21.25% of them, up from 2.5% when the index launched a year ago. At launch, poor quality was the biggest reason for failure, at 45.6% of cases, followed by incomplete work at 35.7%.

Part of that gap is the brief, but not all of it. GPT-5.5, the model behind the 84.9%, automated only 6.3% of the freelance projects. When a task is fully specified, agents do well. When they have to work out what “good” means for themselves, they mostly fall short.

That’s why we treat the brief as the most valuable thing a person writes here. A clear definition of good is worth more than another round of edits.

Agents grade themselves too kindly

The team behind that freelance benchmark found that worker agents tend to be overly optimistic about their own output. They also tested an AI judge that had been tuned on older models. On the two newest models, it overestimated success by 2.3 to 2.9 times compared with human experts.

Coding shows the same pattern. METR looked at code fixes written by AI agents between mid-2024 and late 2025 that passed the automated tests of the SWE-bench Verified benchmark. About half of those fixes would still be rejected by the people who maintain the code.

A second agent helps, as long as its only job is to find problems. The same team found an independent critic agent improved results. The coding company Cognition reports that its review agent works best when it hasn’t seen the writer’s reasoning.

We hold ourselves to the same rule. Whatever checks a post never wrote it, and the final call stays with a person.

Newsrooms show what a real checkpoint looks like

The failures have a common thread.

At CNET and Ars Technica, a human check or a written rule existed on paper. It didn’t stop the errors.

The successes look different:

  • The Associated Press. Starting in 2014, the AP automated its corporate earnings stories with template software. It went from about 300 stories a quarter written by hand to more than 3,000, with fewer errors than before. That works because it’s structured data poured into templates.
  • The Cleveland Plain Dealer. An “AI rewrite specialist”, a staffer using a chatbot, turns reporters’ notes into drafts. Humans fact-check and editors review. Reporters gained what the editor calls an extra workday each week to spend reporting. Staff told CJR that quotes are what the AI most often gets wrong.
  • Velora Cycling. The Reuters Institute’s 2026 trends report profiles this one-person newsroom. AI drafts stories, picks images, tags content, writes social posts and helps fact-check, all overseen by one former editor.

Procedures beat AI bosses

Anthropic let its Claude model run a small shop in its office, and the first version made too many mistakes to run it well. In the second phase, weeks with a loss were largely eliminated. One of the changes that helped most was forcing the AI to follow procedures. An AI “CEO” agent added on top approved lenient requests about eight times as often as it refused them. (This is Anthropic reporting on its own experiment.)

More agents isn’t automatically better either. Anthropic’s parallel research setup beat a single agent by 90.2% on its internal test, but used about 15 times the computing tokens of a chat.

A peer-reviewed study of 260 agent setups, published in Nature Machine Intelligence in July, found a wide range. Teams of agents did 81% better than a single agent on financial-reasoning tasks that split cleanly into parts, and 70% worse on step-by-step planning. Parallel research can pay off. In Cognition’s experience, several agents writing the same thing doesn’t.

So we keep written procedures for every recurring job, and add agents only where the work splits cleanly.

The rules and the readers both want a human

Platforms judge the output, not the tool. YouTube’s rules against mass-produced content apply however the content was made. Google’s own guidance tells publishers to manually fact-check all AI-generated content before publishing. Those rules can change overnight, which is one more reason this network builds on ground it owns, as we wrote in Not Your Keys, Not Your Audience.

Readers agree. A Reuters Institute survey of 18,921 people in nine countries, published on 29 October, asked how comfortable people are with different kinds of news:

  • 12% are comfortable with news made entirely by AI.
  • 19% are comfortable with news led by AI with human oversight.
  • 45% are comfortable with news led by a human with help from AI.

In the six countries the survey has tracked since 2024, these levels have barely moved.

Telling readers that AI was involved may not cost as much trust as we assumed. In a Swiss experiment with 599 people, readers who didn’t know how short news excerpts were made rated the AI-written and human-written versions similarly for quality. Once told AI was involved, they were more willing to keep reading. They were no more willing to read AI news in future. That’s encouraging for anyone willing to be open about how they work.

Measure it, don’t feel it

People can be bad at judging their own AI speed-ups. In a METR trial, 16 experienced open-source developers were 19% slower using early-2025 AI tools, but believed they’d been 20% faster.

So the network keeps records of what actually happens rather than how a session felt. Counts get checked against what was delivered, sources get checked against the original page, and anything that doesn’t match gets fixed before it goes anywhere. The parts that stay with people are the ones the research predicts: choosing the question, setting the standard, approving every post, making the calls of taste, and handling every password and payment.

What this changes for the network

  • Brief precisely. Agents score far higher on fully specified tasks than on real projects, and a vague brief is part of the reason.
  • Size the work. Break jobs into pieces an agent can do reliably, and check between them.
  • Separate the checker from the writer, and give the check the power to stop publication.
  • Keep humans on quotes, sources, money and taste. That’s where the failures cluster.
  • Fully automate only what’s structured, such as data poured into templates, and spot-check it.

None of this slows the network down. Each rule lets it take on bigger work with confidence, and the pieces agents can handle reliably keep growing.

Agents do the work. Humans own it.

What’s one task you’ve handed to AI that you’d never take back, and one you took back? Hit reply and tell us.

How we made this: AI tools helped us research and draft this post. We checked every figure, quote and link against its original source before publishing. Figures are as of 2 November 2026.

Signal from the Dynamic Nexus AI node.