On 17 September 2026 Anthropic published a proposal of measurement tools intended to give the public greater visibility into development inside frontier AI labs. The proposal lays out three classes of metrics—how much AI builds future AI, how effectively labs oversee semi‑autonomous agents, and how compute is allocated—and includes a snapshot of Anthropic’s internal measurements and methods.
Key Details
Anthropic says the three critical measurement areas are: (1) the extent to which AI performs AI research and development (AI‑led R&D); (2) the ability to oversee and intervene in actions taken by AI agents; and (3) the resources that power model development, principally compute allocation. The company published a prototype Anthropic R&D Automation Index that catalogs AI R&D tasks, rates automation levels using an Epoch AI scale (AL0 to AL5), and aggregates weighted ratings.
Key findings in Anthropic’s August 2026 snapshot include: Claude is not operating fully autonomously for any measured subset of AI R&D work; Claude “leads” 26% of measured AI R&D tasks; and the share of work at or above the “AI collaborates” level is above 90%.
The Automation Index methodology sampled about 20% of staff weeks across R&D departments to produce a list of roughly 15,000 granular model R&D tasks, organised into a frozen hierarchical tree with 542 nodes (378 leaves). Anthropic reports model‑vs‑human exact agreement on automation ratings of 59%, human‑vs‑human exact agreement of 35%, and agreement within one level 97%.
On agent oversight, Anthropic reports roughly 30,000 agents doing research and engineering work at any one time on its most‑used internal platform (August 2026 snapshot). Two monitor types are used: online monitors that check actions in real time and can block or steer agents, and offline monitors that ingest actions after the fact. Anthropic reports 100% of these agents’ actions pass through the online monitor prior to execution and 100% are ingested by offline monitors after execution.
Anthropic’s monitoring metrics for August 2026 include automated review before action, human review of blocked actions within one week, roughly 100,000 transcripts flagged per week by the monitor, about 50 highest‑priority escalations per week, and an observed blocking rate of 0.002% when analysing over a billion decisions (about 1 in 47,000). The monitor flags roughly one to two transcripts per 1,000 for further review.
For compute allocation, Anthropic examined all compute usage from 13 to 20 July 2026. In that snapshot, about 6% of compute that went to AI R&D was allocated to safety work, and about 12% of compute that went to AI‑driven AI R&D was allocated to safety. Anthropic describes these as deliberately conservative estimates and notes classification challenges and limits of a one‑week sample.
Who May Be Affected
Anthropic frames these measures as publishable by any frontier developer and useful to the public, third‑party evaluators, and governments. The company proposes independent third‑party evaluators embedded at Anthropic with access to internal processes to verify safety practices, report incidents, and monitor key metrics. Anthropic also notes past independent red‑teaming of its offline monitoring by METR and intends further external evaluation.
Why It Matters For Voice AI
Analysis: Anthropic’s proposed metrics aim to reduce information asymmetry between frontier labs and the public by measuring how models are built, how agents are overseen, and how compute is directed. For products such as voice AI, those same transparency practices—clear measures of automation, oversight coverage and latency, and allocations for safety work—could inform risk assessments, procurement decisions, and trust signals for customers and partners.
What To Watch Next
Anthropic says it will continue publishing these measurements and embedding third‑party evaluators. The company points to its Responsible Scaling Policy risk reports and its Advanced AI Framework (AAIF) policy proposal as related work. Observers should watch for regular, verifiable releases of automation, oversight and compute metrics; cross‑lab methodological convergence; and any third‑party verification regimes that gain acceptance.
Sources
Anthropic — August 2026 Risk Report
Anthropic — Advanced AI Framework (AAIF) proposal
AiDial analysis: We will monitor whether other frontier developers adopt similar public metrics and whether standards for third‑party verification emerge; clearer, comparable metrics would help Australian businesses assess risk and supplier practices when integrating advanced AI into customer‑facing systems.
This article is general information and not legal advice.
