iCreatAI News
Skip to content
Trending storyWatching

Together AI Publishes Coding Agent Evaluation Methodology and Two Fixes

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

Together AI has publicly shared its internal coding agent evaluation process: common development tasks are written as prompts, run twice each with three models under three configurations—baseline, documentation MCP, and skills package—and an LLM judge determines the results as pass, incomplete, or fail. According to Together AI, this evaluation exposed two issues, which the company has fixed separately, and it says the related tools will be released next week, along with an instructional blog post. The evaluation conclusions come from its own account, and the effectiveness of the fixes has not yet been externally verified.

AI overview based on source reports · Updated 1 days ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 16
  1. Together AI Blog
    How we make Together AI easy for coding agents to use

    Together AI describes how it built an agent-evaluation system that runs common developer tasks against baseline, MCP, and skills configurations using multiple models, scoring runs with an LLM judge. The evals surfaced a mislabeled CLI sample and a skill file linking to a marketing page instead of the pricing table, plus an API error message that pointed agents to a nonexistent route; fixes took the GPU cluster task from 4 of 15 passes to 100%. The company also merged 14 separate skills into one unified skill, which initially dropped the pass rate to 75% before fixes brought it to 97.2%, and it plans to open-source its eval package next week.

Activity for this story

Current comparable activity
3
Peak comparable activity
6Nov 17 02:00
Change in the past 24 hours
-49%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.