Airbnb runs coding agents with cost per task
Understand this story
Airbnb shared how it operates its coding agent AirChat on Amazon Bedrock: it routes model requests through a self-built LLM gateway and uses "cost per task" as the basis for model selection. Cost per task refers to the list-price cost of all tokens an agent consumes to complete one task. Airbnb evaluates each model's score and cost on its own tasks simultaneously, then combines CloudWatch metrics on latency, throttling, and token usage to calibrate the default model and the reasoning-effort cap.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogHow Airbnb uses cost per task to operate coding agents on Amazon Bedrock
Airbnb runs its AirChat coding agents through an LLM gateway on Amazon Bedrock and evaluates each model on its own engineering tasks, recording a score and a cost per task for every model and reasoning effort to set defaults and effort ceilings. After rollout, CloudWatch runtime metrics and gateway usage records are compared with eval results; one model that scored well in evals drove several times higher weekly spend per user in production, and moving its heavy users to a newer model cut that group's spend on its main model to about a seventh. The post also covers latency and throttling diagnosis, quota reservation and output-token burndown, and attributing spend by team, application, or feature.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 19 07:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.