Lambda Durable Functions: Checkpoints and Retries

A 90-minute invocation is a long time to be one bad network blip away from starting over. That is the problem Lambda durable functions exist to solve, and pairing them with the new 5,400-second timeout on Managed Instances is where long-running serverless stops being a party trick and starts being an operable system. This is the reliability engineering half of the story: checkpoints, idempotency keys, retry budgets, and the line where you should still reach for a state machine.

Two timeouts, and why lambda durable functions need both

The first thing to get straight is that there are now two different clocks, and confusing them produces very confident, very wrong capacity math.

The function timeout bounds a single invocation. On Managed Instances, for asynchronous and supported event source mapping invocations, that can be up to 90 minutes. The execution timeout bounds a whole durable execution — every step, every checkpoint, every resumption, measured as total elapsed wall-clock time. For durable functions invoked asynchronously, that can run up to a year.

So a durable execution is a sequence of invocations, each of which is individually capped at the function timeout, stitched together across a window that can span months. One step exhausting its 90 minutes does not end the execution. The execution exhausting its window does.

Checkpoint and replay, without the hand-waving

Durable functions recover by replaying. The runtime records the result of completed steps; when an execution resumes after a failure, the code runs again from the top, and completed steps return their recorded results instead of doing the work a second time. Execution continues from the first step that has no recorded result.

That model imposes one rule on your code, and it is not optional: everything outside a checkpointed step must be deterministic. A random value, a fresh timestamp, or a UUID generated in the main body of your handler will be different on replay, and your reconstructed state will quietly diverge from the state you saved. Generate that kind of value inside a step so its result gets recorded, or derive it from the event payload.

Structurally, a long media or ETL job wants to look like this — coarse steps, each one under the function timeout, each one saving something the next step can pick up:

def handler(event, context):
    job_id = event["jobId"]

    manifest = build_manifest(job_id)      # step 1: cheap, deterministic input
    for part in manifest["parts"]:
        transcode_part(job_id, part)       # step N: each well under 90 minutes
    return assemble(job_id, manifest)      # final step: publish the result

The sizing heuristic: steps should be short enough that losing one is annoying, not expensive. If a single step takes 85 of your 90 available minutes, you have a 90-minute function with no recovery story, and you should split it.

Idempotency is your job, not Lambda’s

Lambda does not guarantee exactly-once execution. It never has. Replay makes that concrete rather than theoretical: a step can be attempted, succeed in its side effect, fail before the result is recorded, and be attempted again. If that step charges a card or writes a row, you now have a duplicate.

Use the execution name as your idempotency key. It is stable across replays of the same execution and unique across different ones, which is exactly the property you need — a payload hash is not, because two legitimate identical requests should not collapse into one. The Powertools idempotency utility handles the storage, the in-progress locking, and the response replay against a DynamoDB table:

from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer, IdempotencyConfig, idempotent_function,
)

persistence = DynamoDBPersistenceLayer(table_name="job-idempotency")
config = IdempotencyConfig(
    event_key_jmespath="executionName",
    expires_after_seconds=172800,
)


@idempotent_function(data_keyword_argument="record", config=config,
                     persistence_store=persistence)
def publish_result(record):
    return write_to_system_of_record(record)

Set expires_after_seconds longer than your worst-case execution. An idempotency record that expires while the execution it protects is still running is not protection; it is a delayed duplicate. For executions measured in weeks, that table’s TTL needs to be sized accordingly, and you should be honest about whether DynamoDB TTL granularity is tight enough for your use case.

Retry budgets and where failures land

For asynchronous invocations, the configured retry policy applies — two retries by default — and terminal failures route to your dead-letter queue or on-failure destination. That default was written for functions that ran for seconds. At 90 minutes a shot, three attempts is four and a half hours of dedicated capacity spent on one event that was never going to succeed.

Two adjustments worth making. First, separate the retryable from the terminal in your code, and fail fast on the terminal ones. A missing S3 object, a malformed payload, or a 403 will not be fixed by another 90 minutes. Second, tune MaximumRetryAttempts deliberately per function instead of inheriting the default:

aws lambda put-function-event-invoke-config \
  --function-name long-job-processor \
  --maximum-retry-attempts 1 \
  --maximum-event-age-in-seconds 21600 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:111122223333:job-dlq"}}'

Then make the DLQ useful. A dead-letter queue full of raw payloads with no execution context is a graveyard, not a recovery tool. Include the execution name, the last completed step, and the failure reason in the on-failure destination payload so redrive is a decision rather than an archaeology project. And monitor DLQ depth with an alarm that wakes someone — a 90-minute job failing silently three times a day is an expensive way to produce nothing.

For SQS-backed work, partial batch failure reporting does the same job at batch granularity: a failed record does not drag its successful siblings back through another long invocation. The error handling reference has the response contract.

When Step Functions is still the right answer

Durable functions cover a lot of what people used to build state machines for, so here is the honest dividing line.

Reach for AWS Step Functions when the orchestration itself is the artifact — when non-engineers need to see the workflow, when auditors need a per-step execution record, when steps span multiple services or accounts with different IAM boundaries, when you need a human approval gate, or when you want retry and catch behavior declared in the workflow rather than buried in application code. Fan-out across thousands of parallel branches with Map state is also still its home turf.

Stay in durable functions when the logic is one team’s code, the steps are a sequence rather than a graph, and expressing it as a state machine means translating a plain loop into JSON. That was always an awkward tax to pay, and it was mostly paid because a function could not run long enough. It can now, as the announcement makes clear.

The wrong reason to pick either one is momentum. Pick based on who needs to see the workflow, and how much you would lose if a step failed at minute eighty-nine.

At Enkompass (enkompass.net), this is our home turf. We build event-driven, serverless systems that scale to zero and bill like it — with the idempotency, checkpointing and DLQ design that keeps a long-running job from becoming a long-running incident. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.