The safest way to migrate a Glue job to 6.0 isn’t to flip the version on your production job and hope. It’s to clone it, run the clone on the new runtime next to the original, fix whatever breaks, and only then cut over — with actual DPU-hour and runtime numbers in hand to prove the 30% price cut is really showing up in your bill. This tutorial walks through exactly that, end to end.
Prerequisites
- The AWS CLI v2 configured with an IAM identity that has
glue:GetJob,glue:CreateJob,glue:UpdateJob,glue:StartJobRun,glue:GetJobRuns, andce:GetCostAndUsagepermissions - An existing AWS Glue job currently running on Glue 4.0 or 5.0
- The job’s PySpark script accessible in S3 (you’ll need to edit a copy of it)
- A non-production dataset or a safe subset of production data to test against
- About an hour of uninterrupted time — most of this migration is waiting on job runs, not writing code
Steps to migrate a Glue job to 6.0
-
Pull the existing job definition. Before cloning anything, get the full job spec so you know exactly what you’re working with — worker type, DPU count, script location, arguments.
aws glue get-job --job-name daily-orders-etl \ --query "Job.{Version:GlueVersion,Workers:NumberOfWorkers,Type:WorkerType,Script:Command.ScriptLocation}" -
Clone the job as a new job pointed at Glue 6.0. Never bump the version on the original job directly — create a parallel copy so the production job keeps running untouched while you validate.
aws glue create-job \ --name daily-orders-etl-glue6-test \ --role arn:aws:iam::123456789012:role/GlueETLRole \ --glue-version "6.0" \ --command '{ "Name": "glueetl", "ScriptLocation": "s3://my-etl-scripts/daily-orders-etl-v6.py", "PythonVersion": "3" }' \ --default-arguments '{"--enable-metrics":"true","--job-language":"python"}' \ --worker-type G.1X \ --number-of-workers 5Copy the original script to
daily-orders-etl-v6.pyfirst — don’t point the new job at the same script file you’re about to edit for compatibility fixes. -
Run the clone and let it fail loudly. Kick off a run against your test dataset. Expect it to fail the first time if the script has any Scala UDF dependencies or Python 3.13-incompatible library calls — that’s the point of testing here instead of in production.
aws glue start-job-run --job-name daily-orders-etl-glue6-test -
Fix the Spark 4.1 / Python 3.13 breaking changes. The most common failures on this jump: custom JARs compiled against Scala 2.12 (recompile against 2.13), implicit type casts that Spark 4.1’s stricter ANSI SQL mode now rejects (add explicit
CAST), and deprecated Python stdlib usage that Python 3.13 removed outright.# Before (fails under Spark 4.1 ANSI mode): df.withColumn("amount_cents", col("amount") * 100) # After (explicit cast, passes): df.withColumn("amount_cents", (col("amount") * 100).cast("int"))Run the built-in Spark upgrade agent first to catch most of this before you burn a job run finding it manually — full detail on what it flags is in the official Glue 6.0 migration guide:
aws glue start-job-run \ --job-name daily-orders-etl \ --arguments '{"--job-run-mode":"UPGRADE_ANALYSIS"}' -
Rerun the fixed job on Glue 6.0 and let it complete cleanly. Once the script fixes are in, rerun and confirm success and correct output before you look at cost at all.
aws glue start-job-run --job-name daily-orders-etl-glue6-test aws glue get-job-runs --job-name daily-orders-etl-glue6-test \ --query "JobRuns[0].{State:JobRunState,DPUSeconds:DPUSeconds,ExecutionTime:ExecutionTime}"
-
Compare DPU-hours and runtime against the Glue 4.0/5.0 baseline. Pull the same metrics from a recent production run of the original job and line them up side by side.
aws glue get-job-runs --job-name daily-orders-etl \ --query "JobRuns[0].{State:JobRunState,DPUSeconds:DPUSeconds,ExecutionTime:ExecutionTime}"Divide
DPUSecondsby 3600 for DPU-hours on each side, and compareExecutionTimein seconds directly. It’s common to see both the DPU-hour count drop (thanks to Arrow-native UDF execution and general Spark 4.1 performance gains) and the per-DPU-hour price drop 30% on top of that — a compounding win, not just a discount. -
Confirm the cost delta in Cost Explorer. Numbers from job runs are useful, but the actual bill in Cost Explorer is the proof.
aws ce get-cost-and-usage \ --time-period Start=2026-08-01,End=2026-09-01 \ --granularity MONTHLY \ --metrics "UnblendedCost" "UsageQuantity" \ --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["AWS Glue"]}}' \ --group-by Type=DIMENSION,Key=USAGE_TYPE \ --output table -
Cut over the production job. Once the clone has run cleanly against a realistic dataset with numbers you trust, update the original job in place rather than leaving two jobs running side by side indefinitely.
aws glue update-job \ --job-name daily-orders-etl \ --job-update '{ "GlueVersion": "6.0", "Command": {"Name": "glueetl", "ScriptLocation": "s3://my-etl-scripts/daily-orders-etl-v6.py", "PythonVersion": "3"} }' aws glue delete-job --job-name daily-orders-etl-glue6-test
Troubleshooting common snags
- Job fails immediately with a class-not-found error. Almost always a Scala 2.12 JAR that needs recompiling against Scala 2.13 — check the
--extra-jarsargument for anything you built in-house. - Output row counts differ between the old and new job. Check for implicit casts or null-handling differences under Spark 4.1’s stricter ANSI mode — a silently-dropped bad row on Glue 4.0 may now throw or behave differently on 6.0.
- DPU-hours look identical, no savings visible. Confirm the Cost Explorer query window only covers post-migration usage — mixing pre- and post-migration days in one query will wash out the delta.
- The clone job runs but takes noticeably longer than the original. Double-check the worker type and count on the new job match the original — a clone created with default settings instead of copied settings from Step 1 will run on fewer workers and look artificially slow next to the production baseline.
What you built
You now have a repeatable, low-risk pattern to migrate a Glue job to 6.0: clone before you touch production, run the built-in upgrade analysis before you go hunting for bugs manually, fix the handful of predictable Spark 4.1 and Python 3.13 breaking changes, and validate both correctness and cost with real numbers before cutting over. Run this same sequence against every job still sitting on Glue 4.0 or 5.0, starting with your lowest-risk batch jobs and working up to anything SLA-bound — and keep the Cost Explorer query from Step 7 handy, because it’s the fastest way to prove to anyone asking that the 30% price cut actually landed in your account. Full reference material on job run metrics lives in the AWS Glue job monitoring docs, and current per-DPU-hour rates are always on the Glue pricing page.
At Enkompass (enkompass.net), this is our home turf. We’ve re-architected data pipelines and cut clients’ ETL bills by six figures without a single downstream consumer noticing. If you’d rather have seasoned cloud people handle it instead of learning the hard way at 2 AM, let’s talk. Reach out through our contact page or call us directly at +1-828-436-5667 — we’ll help you turn this from a someday-project into a done-deal.