The HyperPod Inference Gateway is AWS admitting what every team running LLMs on Kubernetes already learned the painful way: a round-robin Service in front of GPU pods is a coin flip with extra steps. One pod is chewing on a 30K-token prompt with its KV cache at 95 percent, the pod next to it is idle, and kube-proxy cheerfully sends the next request to the busy one. This post walks the actual data path, layer by layer, so you know what you are running before you run it.
What the HyperPod Inference Gateway installs
The gateway ships inside the existing amazon-sagemaker-hyperpod-inference EKS managed add-on, version v2.0.0-eksbuild.2 or later, enabled with an inferenceGateway block next to the inferenceOperator block in the add-on configuration. The HyperPod Inference Gateway documentation lists what lands in the cluster: a gateway controller in the hyperpod-inference-system namespace, a GatewayClass named inference-gateway, and the InferenceGatewayConfig CRD (short name igwc). It leans on cert-manager for auto-issued TLS and on the AWS Load Balancer Controller for the load balancer in front.
kubectl rollout status deploy/inference-gateway-controller \
-n hyperpod-inference-system --timeout=150s
kubectl get crd inferencegatewayconfigs.inference.sagemaker.aws.amazon.com
kubectl get gatewayclass inference-gatewayNothing exists on the request path until you create an InferenceGatewayConfig. That resource is the real unit of deployment: one per gateway, namespaced, reconciled by the controller into the three runtime layers below.
Layer 1: Envoy terminates and holds the door
The front is Envoy Gateway, an L7 proxy that terminates HTTPS and exposes one private endpoint per cluster. Leave spec.tls empty and the controller issues a certificate through cert-manager and imports it into ACM, which is why the certificate issuer IAM role (with the AmazonSageMakerHyperPodInferenceGatewayAccess managed policy) has to exist before you install the add-on. Or pass tls.acmArn and bring your own.
Envoy matters for more than TLS. It is where the other two layers plug in as external processing hooks, and it is where request authentication lives. Read this twice: the gateway has no request-level authentication by default. Unless you set spec.auth.jwt with an issuer and a JWKS URI, anything that can reach the private endpoint can burn your GPUs.
Layer 2: the Body-Based Router reads the JSON
Classic ingress routes on host and path. Every OpenAI-compatible call hits /v1/chat/completions, so path routing is useless for picking a model. The Body-Based Router (BBR) buffers the request body, parses it, pulls the model field, and maps that name to the pool whose scheduler declared the same modelName. The idea comes from the upstream Gateway API Inference Extension project, which defines BBR and the Endpoint Picker as reusable pieces on top of the Kubernetes Gateway API.
Three details bite people:
- Body size.
maxRequestBodyBytesdefaults to, and caps at, 256 MiB. Long-context RAG with inlined documents is fine; base64 image payloads at scale deserve a look. - Unknown model names. If the name does not match a scheduler, the request goes to
defaultBackendwhen you set one. Decide on purpose whether a typo should hit a fallback model or fail loudly. - LoRA adapters. A ConfigMap labeled
inference.networking.k8s.io/bbr-managed: "true"maps adapter names to a base model, so a request formodel: "movie-critique"lands on the pool that serves the base model and its adapters.
BBR runs with two replicas by default in hyperpod-inference-system, and it must be enabled whenever more than one scheduler is configured. Single-model gateways can skip it.
Layer 3: the Endpoint Picker scores pods
Once the pool is known, the Endpoint Picker (EPP) chooses the pod. It scrapes each model server’s Prometheus /metrics endpoint and scores candidates on KV cache utilization, queue depth, prefix-cache affinity, LoRA adapter residency, running requests, and predicted latency. Each signal is a weighted scorer; the highest combined score wins. The scheduler field picks the implementation: llm-d (the default, from the llm-d project) or plain epp, and engineType tells it whether to read vllm or sglang metric names.
That is why the version floors matter. The gateway needs vLLM v0.9.2 or later, or SGLang v0.3.5.post1 or later started with --enable-metrics. No metrics, no scoring, and you are back to guessing. EPP pods run in the same namespace as the InferenceGatewayConfig, so a scraping failure shows up there, not in the system namespace.
Here is a config that makes the data-path decisions explicit instead of trusting defaults:
apiVersion: inference.sagemaker.aws.amazon.com/v1alpha1
kind: InferenceGatewayConfig
metadata:
name: prod-gw
namespace: inference
spec:
tls:
acmArn: "arn:aws:acm:us-east-1:111122223333:certificate/abcd-1234"
auth:
jwt:
provider:
name: corp-idp
issuer: "https://idp.example.com"
remoteJWKS:
uri: "https://idp.example.com/.well-known/jwks.json"
bbr:
enabled: true
replicas: 3
maxRequestBodyBytes: 16777216
defaultBackend:
name: llama-8b-svc
port: 8000
schedulers:
- name: llama-8b
modelName: "meta-llama/Llama-3.1-8B-Instruct"
modelSelector:
matchLabels:
app: vllm-llama-8b
targetPort: 8000
scheduler: llm-d
engineType: vllm
routeTimeout: "120s"How the CRD maps to traffic
Each entry in schedulers[] is a pool: a unique modelName (the exact string clients send), a label selector that defines which pods belong, a target port, and its own EPP deployment with two replicas by default. You can define up to 100 schedulers per gateway. Validation enforces unique model names, and weights and configMapRef are mutually exclusive within a scheduler. Models deployed through the HyperPod Inference Operator can opt in with an inferenceGateway.enabled: true block on their InferenceEndpointConfig, which saves you from hand-matching labels.
Failure modes worth rehearsing
The AWS launch post spells out the behavior, and it is sane:
- Pod dies: EPP drops it from candidates and routes to healthy pods; it rejoins when metrics resume.
- Pool saturated: the gateway returns HTTP 429 with
Retry-After. Your clients must honor it, and your autoscaler must react to it. - EPP or BBR unavailable: that is why both default to two replicas. Do not scale them to one to save a few vCPUs.
- Cluster or Region loss: not handled yet. Cross-cluster failover belongs to the Tier-2 Global Inference Router, which is roadmap, not product.
When something looks wrong, trace one request end to end and check the scorer’s view:
kubectl get igwc -n inference prod-gw -o yaml # status conditions
kubectl get pods -n hyperpod-inference-system -l app.kubernetes.io/component=bbr
kubectl logs -n inference deploy/llama-8b-epp --tail=50
curl -s https://GATEWAY_ENDPOINT/v1/chat/completions \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"ping"}]}'Label and deployment names in the last block vary by release; list pods in both namespaces first and adjust. Turning logLevel up to 4 on a scheduler shows the per-pod scores, which is the fastest way to prove whether routing or capacity is your problem.
What the numbers say
AWS benchmarked against round-robin. On mixed GPU generations, P95 and P99 time to first token dropped about 97 percent with 8 to 50 percent more throughput. Bursty traffic saw 94 to 98 percent lower tail TTFT, shared-prefix workloads 26 to 43 percent. A uniform, steady fleet saw roughly no change, which is the honest result: smart routing pays when pods differ in load or cache state. If yours never do, you were not the target customer.
The What’s New announcement confirms per-cluster routing is available wherever the HyperPod inference add-on is. Understand the three layers, set auth on day one, and the rest is tuning.
At Enkompass (enkompass.net), this is our home turf. We design GPU and inference platforms on Amazon EKS that survive production traffic, from model-aware routing and autoscaling to the auth and observability that keep them standing. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.