AI serving moves to production Kubernetes infrastructure
What changed
The llm-d distributed inference project reached v1.0, with maintainers saying open source AI serving is moving from experiments to production infrastructure[1]. Cloudflare also found that prompt caching dominated its AI review spend, making cache behavior a cost lever[2].
Why it matters for this role
Anthropic's Inference TPM role requires a working understanding of how LLM inference uses hardware[3], and its Compute TPM role requires translating the needs of research or ML teams[4]. Nuro's AI Platform TPM role also lists ML infrastructure experience as required[5].
What to do this week
- Learn:
- This week, read the llm-d v1.0 announcement and write a one-page explainer of how an inference request uses GPU memory, batching, and caching, including where cost comes from.
- Resume angle:
- If you have worked with ML or infrastructure teams, write a bullet showing you coordinated delivery of an ML platform or serving project and tied technical tradeoffs such as cost or capacity to schedule decisions.
- Prepare to answer:
- “Walk me through how LLM inference uses hardware, and how that would shape the plan for a program launching a new model on shared GPU capacity.”
- [1]AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan., The New Stack, Oct 23, 2026news
- [2]Cutting our inference bill by 68% by being obsessive about optimization, Cloudflare Blog, Oct 19, 2026news
- [3]Technical Program Manager, Inference, Anthropic, Oct 27, 2026job posting
- [4]Technical Program Manager, Compute, Anthropic, Oct 27, 2026job posting
- [5]Lead Technical Program Manager, AI Platform, Nuro, Oct 27, 2026job posting