Together AI Blog·· 23 hr agoAI score58
Making open models work for long-horizon agentic workloads
Making open models work for long-horizon agentic workloads
AI summary
Ollama and Together AI explain how agentic workloads are dominated by long input sequences and short outputs, making prefix caching essential, and detail harness-side pitfalls that break the cache—such as late-loaded tools, changing system prompts, reasoning-effort changes, oversized tool results, and subagent design—alongside engine-side needs like cache-aware routing, offloading idle caches, PagedAttention, continuous batching, and separating prefill and decode clusters.
Source: Together AI Blog · together.ai