AWS publishes a tutorial on deploying Switchyard routing on Bedrock
Understand this story
AWS has published a tutorial explaining how to use the AWS CDK to deploy the open-source LLM gateway LiteLLM in the user's own AWS account and load the NVIDIA NeMo Switchyard routing plugin to distribute coding agent requests to appropriate models on Amazon Bedrock. Switchyard is an open-source library whose LiteLLM integration supports two routing modes: Stage routing, which reads tool activity and results from the request history and chooses between two candidate models; and Random routing, which splits traffic by weight for A/B comparison. AWS makes clear that the tutorial is sample code and not for production use; the local gateway still calls Amazon Bedrock, and the related requests and smoke tests will incur billed calls; the runner executes model-written Python code on the user's machine.
AI overview based on source reports · Updated 15 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogRoute coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard
This AWS sample deploys LiteLLM with the open source NVIDIA NeMo Switchyard routing plugin on Amazon ECS Fargate via AWS CDK, so coding agents can send requests to a model group that Switchyard routes between a capable and an efficient model on Amazon Bedrock. It covers Stage routing (which reads recent tool activity and results to pick between two candidates) and Random routing (weighted traffic splits for A/B tests), plus a benchmark runner that measures cost per solved task against single-model baselines. The post notes the sample uses switchyard-server 0.3.0 and LiteLLM v1.102.0, is for non-production use, and that its configuration and APIs may change between releases.
Activity for this story
- Current comparable activity
- 7
- Peak comparable activity
- 10Nov 14 07:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.