Building a Serverless DynamoDB Export Pipeline

One filtered export is a CLI command. A DynamoDB export pipeline that ships every tenant’s changes to S3 every hour, keeps Athena in sync, and does not quietly drop a window when something hiccups is an architecture. With filtered exports now available (announced October 1, 2026), you can build that pipeline from managed parts and a few small Lambdas, without a Scan job or a stream consumer in sight.

The shape of the DynamoDB export pipeline

Here is the design we keep landing on:

• EventBridge Scheduler fires an orchestrator Lambda on a fixed cadence.
• The orchestrator reads a tiny control table (also DynamoDB) that tracks, per export stream, the last successfully exported window.
• For each tenant or slice, it issues an incremental filtered export covering the next window, with an idempotent client token.
• An EventBridge rule on the export’s manifest landing in S3 triggers a finalizer Lambda that copies data into a partitioned layout and advances the watermark.
• Glue Data Catalog plus Athena partition projection makes new data queryable without crawlers.
• S3 lifecycle tiers old partitions to Glacier.

Everything is serverless, and every component is boring. That is the point. The only stateful piece you own is the control table, and it holds one small item per export stream. Compare that with the classic alternative, a fleet of Scan workers with checkpoint files and a cron box someone forgot to patch.

Scheduling and window bookkeeping

EventBridge Scheduler is the right trigger here: it has built-in retries, a DLQ, and does not need a rule-per-schedule hack. The interesting logic lives in the orchestrator.

The control table holds one item per export stream, keyed by something like EXPORT#tenant-acme, with a watermark (the last exported ExportToTime) and an inFlight export ARN. The rules:

• Never compute the window from “now minus one hour”. Compute it from the watermark, so a missed run catches up instead of leaving a hole.
• Derive the client token from the stream key and window start. If Scheduler retries the invocation, DynamoDB sees the same token and does not start a duplicate export.
• Set inFlight with a conditional write so two overlapping invocations cannot both launch the same window.

import hashlib, boto3, json, os
from datetime import datetime, timedelta, timezone
ddb = boto3.client("dynamodb")
ctl = boto3.resource("dynamodb").Table(os.environ["CONTROL_TABLE"])
WINDOW = timedelta(hours=1)
def start_window(tenant, table_arn, bucket):
    item = ctl.get_item(Key={"stream": f"EXPORT#{tenant}"})["Item"]
    start = datetime.fromisoformat(item["watermark"])
    end = start + WINDOW
    if end > datetime.now(timezone.utc) - timedelta(minutes=15):
        return None  # window not closed yet
    token = hashlib.sha256(f"{tenant}|{start.isoformat()}".encode()).hexdigest()[:36]
    resp = ddb.export_table_to_point_in_time(
        TableArn=table_arn,
        S3Bucket=bucket,
        S3Prefix=f"raw/tenant={tenant}/window={start:%Y%m%dT%H}/",
        ExportFormat="DYNAMODB_JSON",
        ExportType="INCREMENTAL_EXPORT",
        IncrementalExportSpecification={
            "ExportFromTime": start, "ExportToTime": end,
            "ExportViewType": "NEW_AND_OLD_IMAGES"},
        FilterSpecification={
            "KeyConditionExpression": "#pk = :pk",
            "ExpressionAttributeNames": {"#pk": "pk"},
            "ExpressionAttributeValues": {":pk": {"S": f"TENANT#{tenant}"}}},
        ClientToken=token,
    )
    ctl.update_item(
        Key={"stream": f"EXPORT#{tenant}"},
        UpdateExpression="SET inFlight = :arn",
        ConditionExpression="attribute_not_exists(inFlight)",
        ExpressionAttributeValues={":arn": resp["ExportDescription"]["ExportArn"]})

The filter is a pure key condition on the tenant partition, so each export reads, and bills, only that tenant. Check the SDK reference for your version before you copy the parameter names; the shapes mirror the CLI’s options.

Knowing when an export is done

Exports are asynchronous, so something has to notice completion. Two options: EventBridge on the CloudTrail ExportTableToPointInTime event tells you an export started, which is useful for auditing but not completion. The more reliable signal is the manifest itself. The summary manifest describes a finished export (see the output docs), so turn on EventBridge notifications for the bucket and match on the summary manifest:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": { "name": ["my-export-bucket"] },
    "object": { "key": [{ "suffix": "manifest-summary.json" }] }
  }
}

The finalizer reads the manifest, confirms status with describe-export, copies the listed data files from raw/ into a curated curated/tenant=acme/date=2026-10-16/ layout, advances the watermark and clears inFlight. Why copy? The raw path includes an AWSDynamoDB/<exportId>/ segment that you cannot predict, which makes projection templates awkward. A clean curated prefix pays for itself the first time someone writes a query.

A sweeper on the same schedule handles the unhappy path: any inFlight export older than your SLA gets a describe-export; if it is FAILED, clear it, alert, and let the next run retry the same window. Failed finalizer invocations land in an SQS DLQ, not the void.

Making it queryable

Define one Glue table over curated/ with projection on tenant (injected or enum type) and date (date range). Incremental records carry Keys, Metadata, NewImage and OldImage, so model those as structs. Inserts have only NewImage, deletes only OldImage; a view that takes the latest record per key and drops deletes gives analysts a current-state table without knowing any of this. Expose that view, not the raw table, to BI tools. Nobody in finance should have to learn what an OldImage is.

Seed the curated layout with a one-time full filtered export per tenant, then let incrementals take over from that export time as the first watermark. Skip the seed and your current-state view starts with whatever happened to change in the first hour.

One semantic gotcha worth putting in the README: filters on incremental exports evaluate the latest image, and an item created and deleted inside one window never appears. Filter on keys and those rules stay harmless.

Cost guardrails

• The 10 MB minimum per export adds up when you run hourly exports for hundreds of tiny tenants. Group small tenants into a daily window, or, if your key design allows, give them a shared partition to export together. Check rates on the DynamoDB pricing page.
• Prefer KeyCondition over Filter. Key conditions cut the read you pay for; filter expressions only trim output.
• Lifecycle everything. Expire raw/ after the finalizer copies it, and tier curated/ to Glacier after your query horizon.

The export role needs dynamodb:ExportTableToPointInTime and dynamodb:DescribeExport on the source table, plus S3 write on the raw prefix. Scope it per table; nothing else needs it. And PITR must be on, or the first run fails.

Versus Streams to Firehose and Zero-ETL

DynamoDB Streams into Firehose gives you near-real-time delivery, but you own a consumer, its scaling and its failure modes. Zero-ETL to OpenSearch or Redshift is great when the destination is one of those engines. Scheduled filtered exports win when you want S3 as the system of record, hourly freshness is fine, per-tenant isolation matters, and you would rather not run anything that stays awake.

At Enkompass (enkompass.net), this is our home turf. We build scheduled, watermark-driven DynamoDB-to-S3 pipelines with Glue and Athena for multi-tenant SaaS platforms, including the boring retry and DLQ parts that keep them honest. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.