iCreatAI News
Skip to content
Trending storyUpdating

Together AI Publishes Its Coding Agent Evaluation Methodology

1 reports1 sources18 hr agoUpdated

Understand this story

AI overview

Together AI has published its internal coding agent evaluation methodology: common development tasks are written as prompts and run under three configurations—no configuration, with documentation MCP connected, and with a skill package loaded—with each of three models run twice, and an LLM judge determines whether each run passes, is incomplete, or fails. The methodology exposed two problems, which Together AI says have been fixed. The related tooling will be released next week, accompanied by a usage guide blog post.

AI overview based on source reports · Updated 41 min ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 16
  1. Together AI Blog
    How we make Together AI easy for coding agents to use

    Together AI describes building an agent-evaluation system that runs common developer tasks against baseline, MCP, and skills configurations across three models, using sandboxes, GitHub Actions, a recording proxy, and an LLM judge. The evals surfaced a mislabeled CLI sample and a skill file linking to a marketing page instead of pricing, plus an API error message pointing to a nonexistent route; fixes included two sentences added to the skill and corrected error routing. The company says its unified skill package initially dropped the pass rate to 75% before fixes brought it to 97.2%, and it plans to open-source the eval package next week.

Activity for this story

Not enough continuous observations to show a trend yet.