Turning on the HyperPod Inference Gateway takes an afternoon. Designing a multi-model inference platform on top of it, one that a dozen product teams can share without stepping on each other’s latency, takes decisions. Defaults get you a working endpoint. Tuning gets you a platform. This piece skips the data-path internals and focuses on the knobs: scorer weights, LoRA placement, prompt design that respects caches, metrics, capacity, and what to build now so the upcoming global routing tier is a drop-in rather than a rewrite.
Multi-model inference platform rule one: tune weights per workload
Every scheduler in an InferenceGatewayConfig gets its own Endpoint Picker, and every picker can carry its own weights. The documented defaults are queue: 2, kvCache: 2, prefix: 3, with optional loraAffinity, runningRequests, predictedLatency, and lru (that last one only with the llm-d scheduler). The bias toward prefix affinity is a sensible default for chat. It is wrong for some of your other pools.
A reasonable starting map:
- Agentic and RAG pools with long, shared system prompts and tool schemas: keep
prefixhighest. Cache hits skip prefill, which is where long prompts burn TTFT. - Long-context summarization with unique documents: there is little prefix to reuse, so raise
kvCache. Landing a 60K-token request on a pod near memory saturation triggers preemption and hurts everyone on that pod. - Bursty, short interactive traffic: raise
queueandrunningRequests. Here the enemy is head-of-line blocking, not memory. - Pools mixing GPU generations: add
predictedLatencyso the picker accounts for a slower card finishing the same work later.
schedulers:
- name: support-agent
modelName: "support-agent-v3"
modelSelector: {matchLabels: {pool: qwen-32b-agent}}
weights: {prefix: 5, kvCache: 2, queue: 2}
- name: doc-summarizer
modelName: "summarizer-long"
modelSelector: {matchLabels: {pool: llama-70b-longctx}}
weights: {kvCache: 5, queue: 2, prefix: 1}
- name: autocomplete
modelName: "autocomplete-fast"
modelSelector: {matchLabels: {pool: llama-8b-mixed}}
weights: {queue: 4, runningRequests: 3, predictedLatency: 2, prefix: 1}Treat these as hypotheses. Change one weight at a time, run the same replayed traffic, and compare P99 TTFT. When you outgrow simple weights, configMapRef points a scheduler at a full plugin configuration, the same format the upstream Gateway API Inference Extension uses. It is mutually exclusive with weights, so pick one per scheduler.
LoRA residency is a placement problem
Serving fine-tuned variants as LoRA adapters on a shared base model is the cheapest way to offer many models, but only if requests land where the adapter is already loaded. vLLM holds a bounded number of adapters on the GPU at once (--max-loras), and swapping one in costs time.
The gateway gives you two tools. A ConfigMap labeled inference.networking.k8s.io/bbr-managed: "true" maps adapter names to a base model so the Body-Based Router sends them to the right pool, and a scheduler’s loraAdapters list (up to 50 entries) plus a loraAffinity weight steer requests to pods that already hold the adapter.
apiVersion: v1
kind: ConfigMap
metadata:
name: claims-adapters
labels:
inference.networking.k8s.io/bbr-managed: "true"
data:
baseModel: claims-base-llama-8b
adapters: |
- claims-triage-auto
- claims-triage-home
- claims-fraud-notes
The design rules that follow from this: group adapters that share a base model and similar traffic into one pool; size --max-loras to the hot set, not the full catalog; and give long-tail adapters their own small pool rather than letting them evict hot ones. If a team ships 40 adapters and 3 get 90 percent of traffic, that is two pools, not one.
Design clients for the prefix cache
The prefix scorer can only reward cache hits that your prompts make possible. Prefix caching matches from the first token, so a single changed byte near the top invalidates everything after it. The classic offender is a timestamp or request ID in the system prompt. Publish a client-side contract for every team calling the platform:
def build_messages(tools_block: str, policy: str, history: list, user_msg: str):
# 1. Static, versioned content first: identical bytes on every call
system = f"{policy}\n\n{tools_block}" # tools sorted by name, schemas minified
# 2. Semi-static: conversation history, append-only
# 3. Volatile last: today's date, user context, the actual question
volatile = f"[context] date=2026-10-07 tier=gold\n{user_msg}"
return [{"role": "system", "content": system}, *history,
{"role": "user", "content": volatile}]Sort tool schemas deterministically, version system prompts like code, and never let a library reorder JSON keys between calls. This is the same discipline that makes provider-side prompt caching pay off on Amazon Bedrock; on your own GPUs the payoff is TTFT and fewer replicas.
Observability: measure the scheduler, not just the servers
The stack splits cleanly: pod and pool metrics in Prometheus with Grafana dashboards, cluster and fleet aggregates in CloudWatch, as described in the HyperPod inference observability guide. Three panels pay for themselves: P99 TTFT per model name, queue depth spread across pods in a pool (max minus min tells you whether routing is balancing), and prefix-cache hit ratio per pool.
# P99 time to first token per model (vLLM histogram)
histogram_quantile(0.99, sum by (le, model_name)
(rate(vllm:time_to_first_token_seconds_bucket[5m])))
# Queue imbalance inside a pool: high values mean weights need work
max by (pool) (vllm:num_requests_waiting) - min by (pool) (vllm:num_requests_waiting)Metric and label names shift between vLLM releases, so check your server’s /metrics output before pasting. Also alert on 429 rates at the gateway: they are your earliest signal that a pool is out of capacity.
Capacity planning with mixed hardware
AWS’s benchmarks showed the largest wins on mixed GPU generations: about 97 percent lower tail TTFT and up to 50 percent more throughput versus round-robin. That makes heterogeneous pools a legitimate design choice instead of a liability. Plan capacity per pool from replayed traffic at peak, put the latency-critical pools on the newest cards, and let older generations absorb tolerant work. Scale on queue depth or 429s, not GPU utilization, which reads high even when routing is healthy.
Prepare for the global tier and priority bands
AWS has announced a Tier-2 Global Inference Router for cross-cluster and cross-Region routing with global rate limits (see the launch announcement), canary splits through InferenceModelRewrite, and Critical, Standard and Sheddable priority bands. None of that is shipping yet, but you can design for it now:
- Treat
modelNamevalues as a stable public contract, decoupled from the underlying weights, so canaries can sit behind the same name later. - Tag every caller with a tenant and criticality claim in its JWT today using
requiredClaims; priority bands will need that identity. - Keep pool definitions in Git and identical across clusters, because a global router assumes interchangeable regional pools.
Build for the platform you will need in a year, run the one you have today, and let the metrics tell you which weight to touch next.
At Enkompass (enkompass.net), this is our home turf. We design GPU and inference platforms on Amazon EKS that survive production, including scorer tuning, LoRA pool layout and the observability that proves the routing is working. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.