Architecting Long-Context Workloads on Bedrock

A million-token context window is an invitation to build badly. The tempting design for long-context workloads on GPT-6 Astra is “stop doing retrieval, just send everything” — and for a handful of use cases that’s correct. For most of them it’s a way to turn a $0.04 request into a $6 one while making the answer worse. The architecture question isn’t whether the window is big enough. It’s where the boundary sits between stuffing and retrieving, and what you do about cost on the stuffing side.

When long-context workloads beat chunk-and-retrieve

Retrieval wins when the answer lives in a small, findable slice of a large corpus. “What’s our PTO policy” does not need the handbook, the benefits guide and four years of HR emails in the prompt. Embeddings find the paragraph, a cheap model reads it, everyone goes home.

Full-window inference wins when the question is about relationships that span the corpus, and chunking destroys exactly the signal you need. Three cases show up repeatedly:

• Contradiction hunting. “Where do these documents disagree?” is unanswerable from retrieved chunks, because the retriever has no way to know that section 4.2 matters only because section 11.6 exists. This is the use case AWS leads with in the Astra launch post — financial analysis that reconciles conflicting data, contract review that flags risk-bearing provisions.
• Cross-dependency code investigation. Tracing a behavior through four packages requires following call chains the retriever can’t rank for.
• Agentic loops with accumulated state. A browser-use or computer-use agent that has taken forty steps has forty steps of context, and summarizing it is how agents get stupid.

Everything else, chunk it. A useful heuristic: if you can describe what you’re looking for well enough to write a good search query, retrieval is cheaper and probably more accurate. If the question is “tell me what I should be worried about in here,” send the corpus.

The 272K cliff is an architectural constraint

Astra’s Bedrock pricing steps at 272,000 input tokens, and the step is a doubling. Short context in-Region runs $11 per million input and $55 per million output; above 272K it’s $22 and $82.50. On the global profile it’s $10/$50 versus $20/$75.

The critical detail: the higher rate applies to the entire request. A 275,000-token prompt is billed at the long-context rate for all 275,000 tokens, not at the cheap rate for the first 272K. That makes 272K a real design boundary, not a soft one. A prompt that sits at 280K is paying a 100% premium for 3% more content, and shaving 10K tokens of boilerplate off it cuts the bill in half.

So put the threshold in your code, not in a wiki page nobody reads:

SHORT_CTX_LIMIT = 272_000
HARD_CEILING = 1_000_000          # leave headroom under 1,050,000

def plan_request(token_count: int, workload: str) -> dict:
    """Decide route and flag the expensive path before sending."""
    if token_count > HARD_CEILING:
        raise ValueError(f"{token_count} tokens exceeds the usable window")

    tier = "long" if token_count > SHORT_CTX_LIMIT else "short"

    # Just over the cliff? Trim boilerplate instead of paying double.
    if SHORT_CTX_LIMIT < token_count < SHORT_CTX_LIMIT * 1.15:
        tier = "long-marginal"

    return {
        "tier": tier,
        "model": "global.openai.gpt-6-astra"
                 if workload != "regulated"
                 else "us.openai.gpt-6-astra",
    }

Counting tokens before you send is not optional at this price point. Estimate with a tokenizer locally, then reconcile against the usage block the API returns and alert on drift.

Prompt caching lives on a different endpoint

Here’s the trade that shapes the whole design. The bedrock-runtime endpoint gives you Converse and therefore Guardrails, plus vision and streaming — but no prompt caching. The bedrock-mantle endpoint at https://bedrock-mantle.us-west-2.api.aws/openai/v1 gives you prompt caching (Responses API only) and server-side tool use, but it’s us-west-2 in-Region only, and it has no Converse and therefore no Guardrails.

For long-context workloads that hit the same corpus repeatedly — a legal team asking twenty questions of the same 400K-token data room — caching is the difference between $8 and something close to $1 per query. Cache reads are $1.10 per million in-Region against $11 for fresh input. Cache writes cost $13.75, so the first request is slightly more expensive and every subsequent one is dramatically cheaper.

from openai import OpenAI

# base_url pointed at the bedrock-mantle endpoint,
# api_key = Bedrock API key
mantle = OpenAI()

def ask_cached_corpus(corpus: str, question: str):
    response = mantle.responses.create(
        model="openai.gpt-6-astra",   # in-Region only, no CRIS profile
        input=[
            {"role": "system", "content": corpus},
            {"role": "user", "content": question},
        ],
    )
    print(response.usage)
    return response.output_text

Put the stable corpus first and the volatile question last. That ordering is what makes a cache prefix reusable, and getting it backwards is the most common reason teams report that caching “doesn’t work.”

Routing: global, geo, or both

Cross-Region inference gives you us.openai.gpt-6-astra across the five US and Canada Regions, and global.openai.gpt-6-astra across those plus five EU Regions, six in APAC and São Paulo. Global buys more capacity headroom during demand spikes and costs roughly 10% less per token.

Most shops should run a two-lane setup: global for internal, non-regulated work, geo for anything touching data with residency commitments. Enforce it with an IAM condition on the inference profile ARN rather than trusting every service team to pick correctly, because the cheap option being the loose one is a policy failure waiting to happen. Note that neither profile applies on mantle — that endpoint is in-Region only, so a cached workload is a us-west-2 workload, full stop.

Budgets, circuit breakers and the 128K output cap

Turn on model invocation logging and build your token accounting on it. CloudTrail proves a call happened; invocation logs tell you what it cost. A Logs Insights query gets you a per-workload burn rate in about a minute:

aws logs start-query \
  --log-group-name /aws/bedrock/astra \
  --start-time $(date -d '24 hours ago' +%s) \
  --end-time $(date +%s) \
  --query-string 'fields @timestamp, modelId, input.inputTokenCount as inTok, output.outputTokenCount as outTok
    | stats sum(inTok) as total_in, sum(outTok) as total_out, count(*) as calls by modelId'

Wire an alarm on that metric to a circuit breaker: a Lambda that detaches the Bedrock invoke permission from the workload role when a daily token budget is blown. Blunt, effective, and far better than discovering the problem on the invoice. Pair it with AWS Budgets and Cost Anomaly Detection for the slower-moving version of the same signal.

One last constraint that bites agentic designs: output is capped at 128,000 tokens per response. That’s generous until an agent tries to emit a full refactored file set in one turn. Design loops that emit diffs and plans rather than whole artifacts, checkpoint state between turns, and remember output tokens cost five times what input tokens do. The model card has the current limits if any of this shifts.

At Enkompass (enkompass.net), this is our home turf. We’ve architected production Bedrock workloads where the token budget was a design input rather than a postmortem topic. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.