iCreatAI News
Back to main site
Skip to content

Content types

Evaluation Benchmarks Latest updates

Who is actually stronger: ongoing records of benchmark results, evaluation methodology controversies, and leaderboard changes.

27 条精选Past 30 days: 25 updatesTotal: 189 updates

Updated

精选归档 · 第 2 页

Oct 29Thu 21–27 updates
  1. Artificial Analysis Articles82

    Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

    Google's Gemini 4 Carbon tops the Artificial Analysis Intelligence Index v5 with a score of 59, leading 4 of 9 evaluations, and ranks second to Claude Opus 5.5 on the Coding Agent Index with 59 points. It scores 1609 Elo on AA-Briefcase (up 126 over Gemini 4 Argon) and 54 on AA-Omniscience v1.1, at $7.76 per Intelligence Index task on standard $4/$20 pricing. The model is being rolled out to selected users and is not yet publicly available, with a 50% launch discount and no confirmed end date.

    Why it matters: The piece gives a detailed benchmark breakdown of an unreleased Google model, useful for tracking how Gemini's coding and agentic performance compares with Claude Opus 5.5 and GPT-6 Astra.

Oct 28Wed
  1. Hacker News AI78

    AI Can Draw Blueprints Now

    An architect tested 13 older and newly released AI models on producing construction document sets from concept sketches, finding that Anthropic's Claude Opus 5.5, released September 22, 2026, could produce permit-quality drawing sets for a simple house with skilled guidance and redlining. The models generated vector linework by writing Python code rather than using CAD software, with Opus 5.5 taking twelve hours and a few hundred dollars in compute. The author argues the models' rapid improvement signals a shift for architecture akin to what software engineers experienced, with implications for training, fees and documentation demand.

    Why it matters: Offers a first-hand, technically detailed assessment of AI drafting capability and its likely professional and economic consequences.

  2. Artificial Analysis Articles60

    Announcing Artificial Analysis Intelligence Index v5

    Artificial Analysis released Intelligence Index v5, its biggest upgrade since v4, adding FrontierCode 2.0 and Terminal-Bench-Science and upgrading HLE-Diamond, AA-Omniscience v1.1 and CritPt v2, with 60% of the Index now based on private test sets or held-out answers. Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55), while the open weights gap widens from 3 points to 12. It also launched the Progress Index v1, which tracks model capability over time on one continuous scale.

    Why it matters: It details a major benchmark revision and new leaderboard results, useful for readers tracking how frontier and open-weight models are measured.

Oct 27Tue
  1. Latent Space88

    [AINews] Anthropic files for $2T IPO in 2 weeks: 1400% growth rate, 48% of revenue through cloud partners, 85% margin, $1B op profit in Q2, $134B cash in bank and $10B lifetime burn

    Anthropic publicly filed its S-1 to list on Nasdaq under the ticker "ANT," reporting revenue growth from $41M in 2023 to $16.3B in H1 2026, with Q2 revenue of $11.6B (up 1,368% YoY) and Q2 operating profit of $974M. The filing shows an 85% gross margin in H1 2026, 48% of H1 2026 revenue flowing through cloud marketplaces, $112.4B in cash plus $19.3B in short-term investments against roughly $9.7B in implied lifetime burn, and non-cancelable compute commitments extending into the next decade. It also discloses a recent delay in pre-training a new frontier model and describes "Virtual Collaborators" as the forward pitch for over a billion knowledge workers by 2028.

    Why it matters: The piece condenses a 350-plus-page S-1 into the key financials, compute commitments and strategic claims, making it a useful map of Anthropic's economics and open questions.

Oct 24Sat
  1. The Decoder84

    Chinese AI model Kimi K3.1 Pro is closing in on Anthropic's Claude Opus 5.5

    Moonshot AI released Kimi K3.1 Pro, a post-trained successor to Kimi K3 that scores 55 points on the Artificial Analysis Intelligence Index, an 11-point gain that places it fourth behind Claude Opus 5.5 (58), Claude Sonnet 5.5 (56) and Muse Wonder Full (56). Moonshot says the model keeps K3's 2.8 trillion total and 104 billion active parameters with a one-million-token context window, cuts token prices by 33 percent, and shows its largest gains on agentic tasks, while cyber-related training data was stripped for the planned open-weights release. A faster K3.1 Pro Highspeed variant is also available through Kimi.ai, Kimi Work, Kimi Code and the Kimi API.

    Why it matters: It quantifies how close an open-weights Chinese model has come to top proprietary systems and documents Moonshot's deliberate removal of offensive cyber capabilities before release.

Oct 15Thu
  1. Google Research65

    Putting Earth AI to work: Geospatial agents for our planet's biggest challenges

    Google shares a technical report on its experimental Geospatial Reasoning agent, which connects Gemini to Earth AI foundation models and planet-scale data, and reports evaluations across four benchmarks, including two newly released ones (GeoKTools and GeoExpertQueries). The post describes early partner testing with organizations including WHO AFRO, People in Need, GiveDirectly, the Milken Institute and WPP, and announces expanded early access via Ask Google Earth and an experimental agent-to-agent API.

    Why it matters: It offers a concrete look at how a planning-based geospatial agent is evaluated and applied to real public health, disaster and infrastructure workflows.

Oct 5Mon
  1. GitHub Blog · AI & ML61

    ReviewBench: An open benchmark for AI code review

    GitHub introduces ReviewBench, an open benchmark for AI code review built from analysis of 103.9 million GitHub pull requests, containing 219 public pull requests across 19 languages with a multi-source golden set labeled for severity and category. It reports grounded and augmented precision, recall, and F1 metrics, uses Claude Sonnet 5 as the LLM grader under a published rubric, and had its ground-truth findings independently re-labeled by senior engineers with 96.6% agreement. GitHub says ReviewBench's offline predictions for a Copilot code review multi-model ensemble matched a later A/B test, which showed an 8.0% rise in addressed rate, 13.6% in recall, 61% in comment volume, and an 8.0% drop in cost per review.

    Why it matters: It offers a concrete, reproducible methodology and public dataset for comparing AI code reviewers, plus evidence on how offline scores tracked a production experiment.