Together AI Publishes Its Coding Agent Evaluation Methodology
Understand this story
Together AI has published its internal coding agent evaluation methodology: common development tasks are written as prompts and run under three configurations—no configuration, with documentation MCP connected, and with a skill package loaded—with each of three models run twice, and an LLM judge determines whether each run passes, is incomplete, or fails. The methodology exposed two problems, which Together AI says have been fixed. The related tooling will be released next week, accompanied by a usage guide blog post.
AI overview based on source reports · Updated 41 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Together AI BlogHow we make Together AI easy for coding agents to use
Together AI describes building an agent-evaluation system that runs common developer tasks against baseline, MCP, and skills configurations across three models, using sandboxes, GitHub Actions, a recording proxy, and an LLM judge. The evals surfaced a mislabeled CLI sample and a skill file linking to a marketing page instead of pricing, plus an API error message pointing to a nonexistent route; fixes included two sentences added to the skill and corrected error routing. The company says its unified skill package initially dropped the pass rate to 75% before fixes brought it to 97.2%, and it plans to open-source the eval package next week.
Activity for this story
Not enough continuous observations to show a trend yet.