Book a call
,

Aurora PostgreSQL Iceberg Queries: Inside Embedded DuckDB

Jorge Vazquez
Aurora PostgreSQL database with an embedded DuckDB engine reading Iceberg and Parquet files directly from an S3 data lake

Aurora PostgreSQL Iceberg support went GA on September 30, and it is not another “export your lake into Postgres” feature. AWS bolted DuckDB directly into the Postgres server process, so a foreign table pointing at Iceberg or Parquet in S3 gets planned by Postgres and executed by a real columnar engine, with no copy and no sidecar cluster. Here is how the machinery fits together, what you have to wire up, and where the edges are.

What actually shipped

The feature lives in a new extension, aurora_analytics, available on Aurora PostgreSQL 17.11+ and 18.6+. Per the AWS launch post, it is available in all commercial Regions plus GovCloud (US), and there is no extra charge beyond Aurora compute and the S3 requests you generate. No license line item, no new service to babysit.

The engine underneath is DuckDB, the in-process analytical database that quietly became everyone’s favorite Parquet reader. Embedding it in-process matters: there is no network hop between Postgres and the analytics engine, nothing to install, and a query that joins a foreign table to a regular heap table runs inside one backend. That backend can even see your session’s uncommitted writes, which no external warehouse will ever do for you.

Foreign tables are read-only. INSERT, UPDATE, DELETE, TRUNCATE and COPY FROM against them are rejected. Writes to the lake still belong to Glue, Spark, Athena or whatever already owns your Apache Iceberg tables.

The plumbing: role, feature name, parameter

Three switches have to be on before the extension does anything useful. First, an IAM role trusted by rds.amazonaws.com with read access to your data: s3:GetObject (plus s3:ListBucket for Parquet directories), glue:GetTable on the catalog, database and table ARNs for Glue-registered tables, and the s3tables:Get* actions for S3 Tables. Second, that role attached to the cluster under the AuroraAnalytics feature name. Third, the aurora_analytics.enabled cluster parameter set to true. It defaults to false, which is the step people skip and then spend an hour reading CloudTrail.

aws rds add-role-to-db-cluster \
  --db-cluster-identifier orders-prod \
  --feature-name AuroraAnalytics \
  --role-arn arn:aws:iam::123456789012:role/AuroraAnalyticsRole

aws rds modify-db-cluster-parameter-group \
  --db-cluster-parameter-group-name orders-prod-cpg \
  --parameters "ParameterName=aurora_analytics.enabled,ParameterValue=true,ApplyMethod=immediate"

The parameter is dynamic, so no reboot, unless you are still on the default parameter group and have to swap in a custom one first. Clusters in private subnets also need an S3 gateway endpoint and, for Glue, a Glue interface endpoint with private DNS. The prerequisites page has the minimum-privilege matrix. Then, per database, CREATE EXTENSION aurora_analytics;, which creates the wrapper and a foreign server named aurora_analytics_server.

Foreign tables vs IMPORT FOREIGN SCHEMA

You have two ways to expose lake data. CREATE FOREIGN TABLE handles one table at a time, and if you leave the column list empty, the schema is inferred from Parquet or Iceberg metadata. IMPORT FOREIGN SCHEMA walks a Glue database or S3 Tables namespace and creates a foreign table for every entry in a single transaction. One bad table (unsupported type, missing permission) rolls back the whole import, so use LIMIT TO or EXCEPT when the catalog is messy.

-- Raw Parquet prefix in S3: format required
CREATE FOREIGN TABLE lake.clickstream ()
SERVER aurora_analytics_server
OPTIONS (location 's3://acme-lake/clickstream/', format 'parquet');

-- Iceberg via Glue ARN: format auto-detected, pinned to a point in time
CREATE FOREIGN TABLE lake.orders_eom ()
SERVER aurora_analytics_server
OPTIONS (location 'arn:aws:glue:us-east-1:123456789012:table/sales/orders',
         timestamp '2026-09-30 23:59:59');

-- Every table in a Glue database, minus the junk
IMPORT FOREIGN SCHEMA sales
  EXCEPT (tmp_backfill)
  FROM SERVER aurora_analytics_server
  INTO lake
  OPTIONS (location 'arn:aws:glue:us-east-1:123456789012:catalog');

Rule of thumb: IMPORT FOREIGN SCHEMA for anything catalog-managed, hand-written CREATE FOREIGN TABLE for one-off Parquet drops and for pinning an Iceberg snapshot or timestamp. Note the timestamp is resolved when the table is created, not on every query. When the upstream schema evolves, aurora_analytics_refresh_foreign_table('lake.orders', false) previews the drift and true applies it.

PostgreSQL planner handing a query to a columnar engine that reads selected byte ranges from Parquet files in Amazon S3

Catalog options for Aurora PostgreSQL Iceberg tables

The location option accepts more than S3 paths, and the choice decides how Iceberg metadata gets resolved:

  • Raw S3 URI. For Iceberg, the path must contain a version-hint.text or point straight at a metadata.json. Fine for experiments, fragile for anything another engine is committing to.
  • AWS Glue Data Catalog ARN. The sane default. Format and Region are inferred, and Iceberg resolves the current snapshot through the catalog.
  • Amazon S3 Tables. Either the table ARN directly or through the s3tablescatalog Glue path. Good if you want AWS handling compaction for you.
  • Iceberg REST catalogs. There is no direct IRC client. You federate the external catalog into Glue and reference it with a Glue ARN, which also brings in lakeformation:GetDataAccess on the role.

Iceberg resolves exact file paths from manifests, so it does not need s3:ListBucket. A query can read data in a different Region than the cluster, but not from two Regions in the same query, and S3 Multi-Region Access Points are out.

What runs in Postgres vs DuckDB

Postgres parses and plans. When a query touches a foreign table, the planner tries full query pushdown first: joins, aggregates, CTEs, ORDER BY and LIMIT all get rewritten into DuckDB SQL and executed columnar, including joins against local Aurora tables. If something in the query is not pushdown-safe, it falls back to table-scan pushdown: DuckDB still does the scan with filters and column projection, and Postgres does the rest row by row. EXPLAIN VERBOSE tells you which one you got.

EXPLAIN (VERBOSE, COSTS OFF)
SELECT c.tier, count(*)
FROM lake.clickstream e
JOIN public.customers c ON c.id = e.customer_id
WHERE e.event_date >= DATE '2026-09-01'
GROUP BY c.tier;
-- One top-level "Custom Scan" with a "Pushdown SQL:" line = full pushdown.
-- Custom Scan nested under Postgres nodes plus an
-- "Unsupported Pushdown Expressions" section = fallback.

Typical fallback triggers are Postgres-only functions such as soundex() or UDFs, regex operators, array_agg(), recursive CTEs, and non-C collations on text comparisons. The execution plan docs list the categories. Tuning those plans is a topic of its own; the important architectural point is that the boundary is per query, decided at plan time, and visible.

Sharp edges worth knowing on day one

  • Type mapping gaps. No JSONB, arrays, ranges, network, geometric or XML types on foreign tables. Set aurora_analytics.skip_unsupported_columns to infer around them instead of failing.
  • Column names. Max 63 characters, and no two columns that differ only by case. Spark-generated tables love to violate both.
  • Collations. Only C and ICU on foreign table columns.
  • Cache is ephemeral. The local read cache on NVMe or EBS is wiped on restart or failover.
  • Serverless v2. Concurrent heavy scans can outrun how fast the instance scales up.

Also worth a look before rollout: the memory model (aurora_analytics.query_mem) and the guidance on isolating analytics on a reader, both covered in the how-it-works docs. Spoiler: do not point a dashboard fleet at your writer and call it architecture.

Where this fits

This is not a warehouse replacement. It is the end of a very specific bit of plumbing: the nightly job that copied cold history out of the lake into Postgres so the app could show “last five years of orders.” If your team already speaks Postgres foreign data wrappers, the learning curve is about one afternoon. If your data lives in well-compacted, well-partitioned Iceberg, results come back fast enough that many of those copy jobs can simply be deleted, which is the best kind of migration.

At Enkompass (enkompass.net), this is our home turf. We’ve ripped out reverse-ETL pipelines for clients by pointing Aurora PostgreSQL straight at their Iceberg and Parquet lakehouse data, with the IAM, networking and catalog work done right the first time. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.

Ready when you are

Let’s map your route to AWS

A free 60-minute architecture review with a senior engineer. No slides, no SDR, just a plan.

Book a cloud review
See the work