Claimed vs. Demonstrated
The claims under adjudication split into three layers that are routinely collapsed in coverage: reasoning capability, autonomous agency, and generalized (near-AGI) intelligence. Each has a different evidentiary status. On reasoning, frontier models post high scores on math and coding leaderboards, but independent audits have found that a substantial share of the most-cited benchmark tasks are themselves broken or gameable. An OpenAI-conducted audit of SWE-bench Verified found that a majority of the hardest tasks had tests that would pass even when the underlying bug was unfixed, and OpenAI's own July 2026 audit of SWE-bench Pro found roughly 30% of public tasks contained overly strict tests, underspecified prompts, or misleading prompts, leading it to retract a prior recommendation to adopt that benchmark. This is not a fringe critique; it is the model developer auditing its own headline metric and finding it unreliable.
On autonomous agency, the most rigorous independently-run measurement is METR's time-horizon tracker, which times how long a task a human expert needs versus how long an AI agent can complete the same task at a given reliability threshold. METR's own record states the 50%-reliability time horizon of frontier agents has been doubling roughly every 7 months since 2019, reaching just over two hours for a GPT-5-class agent by its May 2026 measurement, with METR itself flagging that measurements above 16 hours are unreliable with its current task suite. Critically, that trend describes a narrow band of primarily software-engineering and reasoning tasks scored at 50% reliability — a coin-flip success rate unsuitable for most production use. A related analysis found the task length achievable at an 80% reliability bar is roughly a quarter of the 50%-reliability length, meaning the headline exponential-growth number materially overstates what would count as dependable autonomous execution.
The demo-to-production gap shows up directly in independent field data. The 2026 Stanford AI Index found agent success on the OSWorld computer-use benchmark rose sharply, and real-world task success reached a reported 77.3%, but the same index found autonomous-agent deployment across business functions still in the single digits — benchmark gains leading production adoption by a wide margin. Separately, one analysis compiling 2026 field benchmarking found leading models scoring 80-90% on single-turn tasks but dropping to roughly 18-24% on sustained, multi-step workflows spanning applications, illustrating that per-step reliability compounds unfavorably across long agentic chains.
On the enterprise-value question that near-AGI and autonomous-agent claims are meant to justify, MIT's Project NANDA GenAI Divide report (July/August 2025), based on 150 executive interviews, 350 employee surveys, and analysis of roughly 300 public AI deployments, found that despite $30-40 billion in enterprise investment, 95% of generative AI pilots delivered no measurable P&L impact, with only 5% of integrated pilots producing significant value. That figure measures pilot-level financial impact rather than technical failure, and critics have noted the reading in procurement materials often overstates what the underlying data supports, but no comparably rigorous independent study has yet reversed the core finding. Meanwhile Gartner's 2026 Hype Cycle places generative AI in the Trough of Disillusionment while placing agentic AI, a separate and earlier-stage category, near the Peak of Inflated Expectations — the standard signature of capability outpacing demonstrated deployment value.
On near-AGI or general-intelligence framing specifically, the record contains no independent, reproducible demonstration of generalized autonomous intelligence; it contains narrow, task-specific gains (coding, terminal use, math competition problems) measured under benchmark conditions that labs themselves have found to be partially contaminated or exploitable. Hallucination remains an acknowledged, unresolved property of the current training paradigm rather than a solved edge case: OpenAI's own research attributes persistent hallucination to how pretraining and evaluation incentives reward confident guessing over calibrated uncertainty, and states hallucinations remain a fundamental challenge for all large language models despite continued reduction efforts. The honest maturity boundary sits at reliable narrow-task automation under supervision, not generalized autonomous reasoning.
Claims Ledger (6)
What has been independently demonstrated versus claimed — self-reported and not-yet-evidenced are common, honest verdicts.
“Frontier AI agents can now autonomously complete coding tasks that would take a human expert multiple hours, with the reliable time horizon growing exponentially.”independently replicated
Claimed by: METR (independent AI-safety evaluation organization) and widely cited by AI labs and commentators
METR's own tracker, an independent third-party measurement, documents a roughly two-hour, 50%-reliability time horizon for a GPT-5-class agent as of May 2026, with a documented doubling trend since 2019 — but the metric itself is capped at 50% reliability and METR flags unreliability above 16 hours.
“Leading models achieve near-expert or superhuman performance on coding and reasoning benchmarks such as SWE-bench.”contradicted
Claimed by: Model developers' published benchmark tables, widely repeated in trade press
OpenAI's own audit found a majority of the hardest SWE-bench Verified tasks had tests that would pass even when the underlying bug was unfixed, and a later OpenAI audit found roughly 30% of SWE-bench Pro tasks were broken, leading OpenAI to retract its own prior benchmark recommendation.
“Autonomous AI agents are being deployed at scale across enterprise business functions.”asserted not evidenced
Claimed by: Vendor marketing and agentic-AI product announcements
The 2026 Stanford AI Index, an independent research compilation, finds autonomous agent deployment across business functions still in the single digits even as benchmark scores climbed sharply, and MIT's Project NANDA found only 5% of analyzed enterprise pilots produced measurable P&L impact.
“Current generative AI models are approaching artificial general intelligence.”asserted not evidenced
Claimed by: Assorted lab commentary and media framing
No independently reproduced benchmark or deployment record in the search results demonstrates generalized, cross-domain autonomous intelligence; documented gains are narrow and task-specific (coding, terminal use, competition math), and hallucination is acknowledged by OpenAI's own research as a persistent, unresolved property of current models rather than a solved capability gap.
“Multi-step AI agent workflows are reliable enough for unsupervised production use.”contradicted
Claimed by: Agentic AI product marketing
2026 field benchmarking compiled across the industry shows leading models scoring 80-90% on single-turn tasks but dropping to roughly 18-24% on sustained, multi-step workflows crossing applications, and Gartner has forecast more than 40% of agentic AI projects will be canceled by the end of 2027 over cost and unclear value.
“Hallucination in large language models has been effectively solved or reduced to a negligible edge case.”self reported
Claimed by: Some vendor product marketing framing reduced hallucination rates as near-solved
OpenAI's own published research states hallucinations remain a fundamental challenge for all large language models despite continued reduction efforts, attributing persistence to evaluation incentives that reward confident guessing over calibrated uncertainty — a structural rather than fully resolved issue.
Documented Builders (4)
Only actors whose involvement is publicly documented, with the record named.
METRHigh
Independent AI evaluation organization measuring autonomous agent time-horizon capability across frontier model releases
Basis: METR's publicly documented time-horizon tracker and its March 2025 and January 2026 ("Time Horizon 1.1") published methodology updates
OpenAIHigh
Frontier model developer; conducted and published internal audits of SWE-bench Verified and SWE-bench Pro benchmark validity
Basis: OpenAI's published benchmark audit findings (February and July 2026) and OpenAI's own research paper on why language models hallucinate
MIT Project NANDAHigh
Academic research initiative that produced the GenAI Divide enterprise-adoption study
Basis: The GenAI Divide: State of AI in Business 2025 report, based on approximately 150 executive interviews, 350 employee surveys, and roughly 300 analyzed enterprise deployments
GartnerMedium
Industry research firm publishing hype-cycle placements for generative AI and agentic AI and forecasting agentic-AI project cancellation rates
Basis: Gartner's 2026 Hype Cycle for Generative AI and its April 2, 2026 Hype Cycle for Agentic AI
Barriers (4)
Benchmark validity and contamination — leading coding and reasoning benchmarks contain broken, gameable, or contaminated test itemstechnicalengineering effort
OpenAI's own audits found a majority of the hardest SWE-bench Verified tasks passable without fixing the underlying bug, and roughly 30% of SWE-bench Pro tasks broken, prompting a retracted benchmark recommendation.
Reliability decay on long, multi-step agentic workflowstechnicalfundamental science
2026 field benchmarking found success rates falling from 80-90% on single-turn tasks to roughly 18-24% on sustained cross-application workflows, and METR's data shows the 80%-reliability time horizon is roughly one-quarter of the 50%-reliability horizon, indicating no demonstrated solution yet for compounding step-wise error at scale.
Enterprise integration and workflow-adaptation gap preventing pilots from converting to measurable valueeconomicstructural
MIT's Project NANDA found 95% of analyzed enterprise pilots delivered no measurable P&L impact despite substantial investment, attributing much of the gap to organizational integration and learning failures rather than model capability alone.
Persistent hallucination rooted in training and evaluation incentive structurestechnicalfundamental science
OpenAI's published research attributes hallucination to pretraining dynamics and evaluation incentives that reward confident guessing over calibrated uncertainty, describing it as a fundamental challenge for all large language models rather than a fully solved property.
Watch Signals (4)
A new independent, contamination-audited benchmark suite gains adoption across multiple frontier labs to replace disputed SWE-bench variants
This would indicate the field has repaired the specific measurement-validity problem OpenAI's own audits identified, restoring confidence in headline capability claims
Where to watch: Track benchmark-adoption announcements from major labs and academic evaluation groups such as METR
METR publishes a time-horizon update extending reliable measurement past its current 16-hour ceiling
This would be the clearest independent signal that autonomous agent reliability is genuinely extending toward multi-day task horizons rather than plateauing at the current measured range
Where to watch: Monitor METR's public time-horizons tracker and blog for methodology updates
A follow-up rigorous enterprise-adoption study shows the pilot-to-value conversion rate materially improving beyond the 5% figure MIT's Project NANDA documented
This would be the strongest available evidence that capability gains are finally converting into measured business impact at scale
Where to watch: Watch for follow-on studies from MIT Project NANDA, Gartner, or comparable independent research organizations
Gartner moves agentic AI off the Peak of Inflated Expectations toward its Trough in a future Hype Cycle for Agentic AI update
This institutional signal would mark the point where market expectations begin correcting to match demonstrated deployment reliability
Where to watch: Track Gartner's published annual Hype Cycle for Agentic AI reports
Facts & Figures (12)
The claims behind this analysis, each with its verification status — including what is contested, unverified, or could not be established.
Frontier AI agents can now autonomously complete coding tasks that would take a human expert multiple hours, with the reliable time horizon growing exponentially.
METR's own tracker, an independent third-party measurement, documents a roughly two-hour, 50%-reliability time horizon for a GPT-5-class agent as of May 2026, with a documented doubling trend since 2019 — but the metric itself is capped at 50% reliability and METR flags unreliability above 16 hours.
✓ INDEPENDENTLY REPLICATEDclaimed by METR (independent AI-safety evaluation organization) and widely cited by AI labs and commentators
Leading models achieve near-expert or superhuman performance on coding and reasoning benchmarks such as SWE-bench.
OpenAI's own audit found a majority of the hardest SWE-bench Verified tasks had tests that would pass even when the underlying bug was unfixed, and a later OpenAI audit found roughly 30% of SWE-bench Pro tasks were broken, leading OpenAI to retract its own prior benchmark recommendation.
◑ CONTRADICTEDclaimed by Model developers' published benchmark tables, widely repeated in trade press
Autonomous AI agents are being deployed at scale across enterprise business functions.
The 2026 Stanford AI Index, an independent research compilation, finds autonomous agent deployment across business functions still in the single digits even as benchmark scores climbed sharply, and MIT's Project NANDA found only 5% of analyzed enterprise pilots produced measurable P&L impact.
○ ASSERTED NOT EVIDENCEDclaimed by Vendor marketing and agentic-AI product announcements
Current generative AI models are approaching artificial general intelligence.
No independently reproduced benchmark or deployment record in the search results demonstrates generalized, cross-domain autonomous intelligence; documented gains are narrow and task-specific (coding, terminal use, competition math), and hallucination is acknowledged by OpenAI's own research as a persistent, unresolved property of current models rather than a solved capability gap.
○ ASSERTED NOT EVIDENCEDclaimed by Assorted lab commentary and media framing
Multi-step AI agent workflows are reliable enough for unsupervised production use.
2026 field benchmarking compiled across the industry shows leading models scoring 80-90% on single-turn tasks but dropping to roughly 18-24% on sustained, multi-step workflows crossing applications, and Gartner has forecast more than 40% of agentic AI projects will be canceled by the end of 2027 over cost and unclear value.
◑ CONTRADICTEDclaimed by Agentic AI product marketing
Hallucination in large language models has been effectively solved or reduced to a negligible edge case.
OpenAI's own published research states hallucinations remain a fundamental challenge for all large language models despite continued reduction efforts, attributing persistence to evaluation incentives that reward confident guessing over calibrated uncertainty — a structural rather than fully resolved issue.
○ SELF REPORTEDclaimed by Some vendor product marketing framing reduced hallucination rates as near-solved
METR's time-horizon tracker measured a GPT-5-class agent at roughly 2 hours 17 minutes of human-expert task time at 50% reliability as of its May 8, 2026 update, and explicitly flags that measurements above 16 hours are unreliable with its current task suite.
This is the most rigorous independent metric for agent autonomy, and it caps the reliably-measured autonomy window at hours, not days or weeks, directly contradicting looser 'autonomous agent' framing.
✓ GROUNDED
OpenAI's July 8, 2026 audit of SWE-bench Pro found roughly 30% of public benchmark tasks were broken (overly strict tests, underspecified or misleading prompts), leading it to retract a prior recommendation to adopt that benchmark over SWE-bench Verified.
A model developer auditing and retracting confidence in its own recommended coding benchmark demonstrates that headline reasoning/coding scores are less reliable than leaderboard rankings suggest.
✓ GROUNDED
MIT's Project NANDA GenAI Divide report (2025), based on roughly 150 executive interviews, 350 employee surveys, and analysis of about 300 enterprise deployments, found 95% of generative AI pilots delivered no measurable P&L impact despite $30-40 billion in enterprise investment, with only 5% of integrated pilots producing significant value.
This is the load-bearing evidence that capability claims have not translated into demonstrated business value at scale, regardless of benchmark trajectory.
● YOUR INPUT
The 2026 Stanford AI Index found agent success on the OSWorld computer-use benchmark rose from roughly 12% to about 66%, with real-world task success reaching a reported 77.3%, while autonomous agent deployment across business functions remained in the single digits.
This is the clearest documented instance of the benchmark-versus-deployment gap: large measured capability gains have not yet converted into measured production usage.
✓ GROUNDED
Gartner's 2026 Hype Cycle places generative AI in the Trough of Disillusionment, while its separate Hype Cycle for Agentic AI (published April 2, 2026) places agentic AI near the Peak of Inflated Expectations heading toward its own trough.
The two adjacent technologies are at different, independently-tracked hype-cycle stages, which means autonomous-agent claims should not borrow generative AI's more mature adoption curve as evidence of their own readiness.
● YOUR INPUT
An analysis of METR's own published data found that the task length achievable at an 80% success-reliability threshold is roughly one-quarter the task length achievable at the 50% threshold used for the widely-cited exponential time-horizon trend.
This shows the most-cited autonomy growth curve is calibrated to a coin-flip reliability standard, materially overstating the dependable autonomy window relevant to production use.
✓ GROUNDED