How Airbnb uses cost per task to operate coding agents on Amazon Bedrock
How Airbnb uses cost per task to operate coding agents on Amazon Bedrock
Airbnb runs its AirChat coding agents through an LLM gateway on Amazon Bedrock and evaluates each model on its own engineering tasks, recording a score and a cost per task for every model and reasoning effort to set defaults and effort ceilings. After rollout, CloudWatch runtime metrics and gateway usage records are compared with eval results; one model that scored well in evals drove several times higher weekly spend per user in production, and moving its heavy users to a newer model cut that group's spend on its main model to about a seventh. The post also covers latency and throttling diagnosis, quota reservation and output-token burndown, and attributing spend by team, application, or feature.
Source: AWS Machine Learning Blog · aws.amazon.com