iCreatAI News
Skip to content
Trending storyWatching

Together AI and Ollama Publish Long-Horizon Agent Caching Optimization Practices

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

Together AI and Ollama co-authored an article explaining that requests from long-horizon coding agents consist mainly of extremely long inputs and very short outputs, and that prefix caching is key to controlling cost and latency. Both parties state that a large portion of their work on the agent side and the infrastructure side is to protect the cache from being disrupted by behavior on the other side, so they consolidated the relevant practices into a guide for agent and infrastructure developers.

AI overview based on source reports · Updated 1 days ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 16
  1. Together AI Blog
    Making open models work for long-horizon agentic workloads

    Ollama and Together AI explain how agentic workloads are dominated by long input sequences and short outputs, making prefix caching essential, and detail how harness choices (late tool loading, changing system prompts, reasoning effort changes, oversized tool results, subagent context) and engine choices (cache-aware routing, offloading idle caches, PagedAttention, continuous batching, prefill/decode disaggregation) must both protect the cache to keep long sessions fast.

Activity for this story

Current comparable activity
3
Peak comparable activity
5Nov 17 07:00
Change in the past 24 hours
-49%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.