AutomationBench-AA and the Difference Between Progress and Completion
AutomationBench-AA gives partial credit for completed objectives. Compare that score with full task completion before choosing a workflow agent.
By Yuki Yu
An editorial agent links the right cover, writes the right metadata and attaches both figures. Then it publishes a draft that was supposed to stay private. How much credit did it earn? A product manager counting completed steps and an editor responsible for the live page could give very different answers.
That distinction matters when reading AutomationBench-AA, a workflow evaluation included in the Artificial Analysis Intelligence Index v5 announced on October 27, 2026. Its headline score gives partial credit for objectives achieved on a task that passes its guardrails. A separate measure counts tasks completed in full. A buyer promising unattended delivery should ask for both numbers. Artificial Analysis v5 announcement.
Read first. Decide what an acceptable result looks like in your own application before translating a public benchmark into a purchasing decision. Token360’s AI model evaluation framework.
What does AutomationBench-AA measure?
It measures how much of a business workflow an agent can complete in simulated software. Artificial Analysis describes 657 tasks across finance, HR, marketing, operations, sales and support. A run might require reading one system, finding the correct record in another and updating a third. The grader inspects the resulting application state. Artificial Analysis evaluation methodology.
That last detail changes how to interpret a polished response. “I updated the customer record” sounds equally plausible whether the agent updated the correct account, a similarly named account or neither. A useful check reads the specific record back and compares it with the expected value. If the task also requires preserving a second record, check that one too.
Zapier developed AutomationBench and publishes a leaderboard using full task completion as its official metric. Its description explains how a simulated company starts from a seeded state and how assertions inspect what remains at the end. When two reports use the AutomationBench name, record the benchmark operator and metric label alongside the figure you quote. Zapier’s AutomationBench methodology.
How do the two scores differ?
For the AA headline score, a clean task earns the fraction of its objectives achieved. A guardrail violation makes that task’s score zero. The headline aggregates the task-level results. For the full-completion measure, a task counts only when all of its objectives are achieved and its guardrails hold. AutomationBench-AA score definitions.
The first measure helps a team find a promising assistant for supervised work. If someone already reviews every package, a partially prepared package may save effort. Full completion is closer to the question an operations lead asks before removing a review step. Even that figure needs context. An assistant that fails harmlessly and one that writes to an unauthorized destination can have the same full-completion rate.
What would these metrics look like at a publishing desk?
Consider four hypothetical editorial jobs with four verifiable objectives each. Each job asks an agent to prepare a draft with the assigned category, correct cover, both approved figures and reviewed metadata. The controlling rule keeps the draft unpublished. An objective is credited when a reviewer can find the expected value on the assigned draft.
Job
Final state
Objective credit after guardrail check
Fully complete
A
All four objectives met, draft private
4/4
Yes
B
Metadata missing, draft private
3/4
No
C
Cover and figures missing, draft private
2/4
No
D
All four fields correct, page published
0/4
No
The four task scores are 100%, 75%, 50% and 0%. Their mean is 56.25%. One of four packages is ready as requested, so full completion is 25%. An unchecked field counter would report 13 of 16 objectives, or 81.25%, because it would credit the fields on the published page. That difference is exactly where the publishing constraint enters the calculation.
These are calculated examples using invented outcomes. No model performed the jobs and no CMS records were changed for the example. Four objectives per job keep the arithmetic easy to audit. For jobs of different sizes, averaging each job’s fraction gives equal weight to jobs. Pooling all objectives instead gives more influence to the jobs with more checks. Specify the aggregation before comparing candidates.
Which failures deserve their own column?
Separate missing work, incorrect work and prohibited actions. They may all prevent completion, but each creates a different follow-up job. Missing metadata can be filled in. The wrong cover must be identified and replaced. An unintended publication may require taking down a page and checking which downstream systems copied it.
For a CMS pilot, write guardrails around observable boundaries. Name the permitted site, language, draft ID and publication state. Prepare a second draft with a similar title so a reviewer can detect a write to the wrong record. Ask for evidence of what changed and which neighboring records stayed intact. A successful HTTP response is useful for tracing a call, but the target and final state decide whether the work passed.
Also distinguish a forbidden attempt from a lasting change. A permission layer might block a publish call. The draft remains private, yet the attempted action still matters to a security reviewer. An end-state assertion can verify the former. A tool log can reveal the latter. Report both if your adoption decision depends on the agent respecting the instruction as well as the platform enforcing it. Request-level observability for AI applications.
How does partial progress translate into reviewer time?
Measure the time a reviewer spends getting each package to an acceptable state. A model that leaves one hard publishing decision unresolved can require more time than one that leaves several routine labels blank. The number of objectives alone cannot price the remaining work.
In the example, suppose checking A takes one minute, finishing B takes three, finishing C takes six and investigating D takes twenty. That is thirty minutes of follow-up across four assignments. The fourth job consumes two thirds of the review time. These assumed minutes illustrate the calculation. A production trial should time real reviewers and include time spent finding an error before fixing it.
Record which cost you are comparing. Inference spend per attempted task is straightforward. Cost per fully completed task can be much higher when many attempts stall. For an assisted workflow, include review time and count the packages finally accepted. For an unattended workflow, count the packages that reached the correct state without human repair. Both calculations can be valuable in the same trial.
What should go into a first-party workflow pilot?
Choose one bounded workflow and write down its allowed starting state, required end state and forbidden changes. Use a sandbox with realistic record names and a known reset procedure. Save a snapshot before and after each run so an evaluator can reproduce the judgment.
Give each candidate the same tools and permissions. Record model identifier, reasoning setting where applicable, tool schema, timeout and retry policy. A model that receives an exact draft ID faces a different task from one that must search a catalog. If the production assistant can ask a clarifying question, score that behavior under a separate condition and track whether the question actually resolves the ambiguity.
Start the scorecard with three columns: objectives completed, entire task accepted and guardrails triggered. Add investigation time and tool traces for the failures that matter most. Review representative examples before adopting an aggregate pass threshold. One record updated in the wrong workspace may tell you more about the release decision than several successful formatting changes.
Frequently asked questions
Does a 60% AutomationBench-AA score mean 60% of tasks succeeded?
Only the full-completion measure answers that question. The headline objective score can include credit from partly finished tasks that satisfy their guardrails.
Why compare the Zapier and Artificial Analysis pages?
Zapier explains the original benchmark and its official completion metric. Artificial Analysis explains its independent evaluation and headline scoring. Report the operator, metric and run configuration together.
Can a task with every requested field correct still fail?
Yes. In the publishing example, setting all fields correctly and publishing a draft that had to remain private triggers the controlling guardrail.
Is supervised deployment reasonable with modest full completion?
It depends on the time saved after review, the consequences of missed errors and the permissions available to the assistant. Measure the human work left on each case.
What should I do with tool errors?
Retain the task outcome and label the cause separately. A timeout, a wrong target and a misunderstood instruction each call for a different fix.
Does this example rank models available through Token360?
The editorial case demonstrates scoring arithmetic. To compare routes for your own application, replay a fixed task set with the intended model IDs, permissions and acceptance rules.
Prompt Library
Insights
Insights