Together AI and Ollama Publish Long-Horizon Agent Caching Optimization Practices
Understand this story
Together AI and Ollama co-authored an article explaining that requests from long-horizon coding agents consist mainly of extremely long inputs and very short outputs, and that prefix caching is key to controlling cost and latency. Both parties state that a large portion of their work on the agent side and the infrastructure side is to protect the cache from being disrupted by behavior on the other side, so they consolidated the relevant practices into a guide for agent and infrastructure developers.
AI overview based on source reports · Updated 1 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Together AI BlogMaking open models work for long-horizon agentic workloads
Ollama and Together AI explain how agentic workloads are dominated by long input sequences and short outputs, making prefix caching essential, and detail how harness choices (late tool loading, changing system prompts, reasoning effort changes, oversized tool results, subagent context) and engine choices (cache-aware routing, offloading idle caches, PagedAttention, continuous batching, prefill/decode disaggregation) must both protect the cache to keep long sessions fast.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 5Nov 17 07:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.