Should You Still Partition Delta Tables? When to Switch to Databricks Liquid Clustering

Almost every Delta table we inspect on a new engagement is partitioned by date. Load month, ingestion day, event date. It is the habit an entire generation of data engineers learned, and until recently it was correct.

It is now the wrong default for most tables. Databricks has said so plainly: liquid clustering replaces partitioning and Z-order, and automatic liquid clustering is switched on for new Unity Catalog managed tables. What the announcements skip is the part that actually matters to a team running production workloads: when do you switch a table that already works, what does the conversion cost on ten terabytes, and when does the old way still win?

This article is the decision guide we use on client estates. It covers how liquid clustering works, a concrete decision framework, the migration runbook including the in-place conversion path, and an honest evaluation of the automatic mode. It is written for the engineer who asked the question we see everywhere from Reddit to the Databricks Community forums: "We partition everything by month. Should we stop?"

Want the announcement context? Watch the video above, or keep reading for the decision framework Databricks will not give you.

Why is date partitioning the wrong default now?

Partitioning was never a performance feature. It was a workaround for file listing: physically separating data into directories so engines could skip whole folders. That workaround has three failure modes every practitioner eventually meets.

Small files. Partition by day, then by another column, and the combinatorics explode. A daily partition scheme crossed with a few thousand customers can produce millions of tiny files. Every query pays the file-listing and open-file tax, and your OPTIMIZE jobs spend their lives gluing fragments back together.

Skew. Real data is not uniform. One tenant, one region or one month dominates, and that partition becomes a monolith while thousands of others hold a few kilobytes. Community post-mortems on 300-million-row tables tell the same story: the layout that was supposed to speed things up became the bottleneck.

Rigidity. This is the expensive one. Partition columns are fixed at table creation. When query patterns change, and they always change, the only way out is a full rewrite. Choosing partition keys was a one-way door.

Liquid clustering exists because of that third failure. Clustering keys are metadata, not directory structure: you can change them with a single ALTER TABLE and no data rewrite. The one-way door becomes a corridor.

How does liquid clustering work in Databricks?

Liquid clustering organises data within files using a space-filling curve (a Hilbert curve) rather than segregating it into directories. Rows that are close together in the clustering-key space end up physically close on disk, so queries filtering on those keys skip most files. The effect is what partitioning plus Z-order promised, without either of their constraints.

Three properties matter in practice:

• It is incremental. Clustering happens as part of OPTIMIZE, and only on data that benefits. You are not rewriting the table on every run.
• Keys are changeable. ALTER TABLE orders CLUSTER BY (customer_id, order_date) takes effect for future clustering with no rewrite of existing data.
• It subsumes Z-order. If you are maintaining Z-order jobs today, liquid clustering replaces them. The two do not combine.

Enabling it on a new table is one clause: CREATE TABLE orders (...) CLUSTER BY (customer_id, order_date);

Databricks' own benchmarks (their "8 data layout myths" post from June 2026) claim 22% faster query performance and 27x faster metadata operations versus a well-partitioned baseline. Treat vendor numbers as vendor numbers, but our observation on client tables points the same direction: for selective queries on high-cardinality keys, the difference is not subtle.

Liquid clustering vs partitioning: the decision framework

The question is never "which is better". It is "which does this table need". Here is the framework we apply, table by table.

Switch to liquid clustering when:

• Queries filter on high-cardinality columns (customer ID, device ID, order ID) that partitioning could never serve without exploding file counts.
• The table suffers from skew or small files today. Clustering absorbs skew instead of amplifying it.
• Query patterns evolve. If you have ever regretted a partition key, that table is a clustering candidate by definition.
• You run Z-order maintenance. Liquid clustering does the same job with less ceremony.

Leave partitioning in place when:

• The filter pattern is static, predictable and high-selectivity, and the partition sizes are healthy (roughly 1 GB or more each). A large append-only table always queried by exact date that has run clean for two years is not a problem to solve.
• External readers depend on it. The delta-rs ecosystem (and tools built on it) does not currently support liquid clustering. If Rust- or Python-native readers outside Databricks consume the table, check compatibility before touching the layout.
• The table is small. Under a few hundred megabytes, neither partitioning nor clustering earns its keep; both just add operational noise.

Do nothing when: the table is small, rarely queried, or already fast. Layout migration is real work. Spend it where scans hurt.

One more input belongs in the decision: your medallion layer. Bronze tables that are written once and read sequentially rarely justify clustering. Silver and gold tables carrying selective analytical queries are where the wins live. We covered the layer-by-layer layout logic in our medallion architecture best practices guide.

How do you migrate a partitioned table without a full rewrite?

This is the question the docs answer thinnest and communities ask loudest: does converting a 10 TB partitioned table mean rewriting 10 TB?

Not anymore. On recent runtimes (Databricks Runtime 18.1 and later), you can convert in place: ALTER TABLE orders REPLACE PARTITIONED BY WITH CLUSTER BY (customer_id, order_date);

The conversion changes the table's layout contract immediately; the physical re-layout happens incrementally through subsequent OPTIMIZE runs rather than as one monolithic rewrite. On older runtimes, the path is CTAS into a new clustered table, which does mean a full copy. If you are planning conversions at fleet scale, the runtime version is worth checking first.

The runbook we use:

• Pick keys from evidence, not intuition. Profile the query history: which columns actually appear in WHERE clauses and joins against this table? One to three keys, uncorrelated with each other.
• Convert or create. In-place conversion on Runtime 18.1 and later, CTAS below it. For new tables, cluster from day one; retrofitting is always the more expensive path.
• Run OPTIMIZE and expect a storage bump. Clustering rewrites files, and old files stay until VACUUM clears them. Plan for the table to temporarily approach double its size, and schedule VACUUM accordingly. On a multi-terabyte table this is a real capacity conversation, not a footnote.
• Verify pruning before declaring victory. Check EXPLAIN output and DESCRIBE DETAIL on representative queries. Files-read counts should drop on clustered-key filters.
• Give it days, not minutes. Community experience describes liquid clustering as feeling like "lazy optimization": the layout converges over successive OPTIMIZE runs. Benchmark after the ramp, not before.
• Keep a rollback plan. Until you have verified pruning and cost on production queries, keep the ability to fall back: either the pre-conversion snapshot via time travel windows or the old table until sign-off.

Two ecosystem gotchas worth knowing before you convert anything wired into dbt: the dbt-databricks adapter has an open issue (#1329) affecting streaming tables with liquid clustering, and materialized views on clustered sources can force full refreshes. Neither is a blocker; both are the kind of surprise you want to meet in a plan, not in production.

Is CLUSTER BY AUTO good enough for production?

Automatic liquid clustering (CLUSTER BY AUTO) lets Databricks pick and adjust clustering keys itself, driven by predictive optimization analysing your query patterns. It is now the default for new Unity Catalog managed tables, and the honest answer to "is it reliable" is: yes, within limits you need to know.

The limits, none of which announce themselves:

• It ignores small tables. Below roughly 256 MB, automatic key selection does not engage. The setting sits there doing nothing, silently.
• It needs evidence. Key selection wants a history of pruning-eligible scans (on the order of ten or more). A new table, or one queried rarely, gives it nothing to learn from.
• It depends on predictive optimization. If predictive optimization is not active on the schema, AUTO is a no-op. Verify it is actually running before trusting it; do not assume.

Community sentiment is appropriately sceptical. The most-endorsed advice in the threads we analysed says it directly: "I recommend against using the automatic setting; it's essential to have a solid understanding of your table." We would phrase it more moderately: AUTO is a good default for well-trafficked managed tables where query patterns are stable enough to learn from. For your highest-value tables, pick keys manually from query-history evidence, because you understand intent and AUTO only sees history.

Manual key selection rules that hold up in practice:

• One to three keys. Community testing consistently finds that more keys degrade single-column filter performance. Fewer, better-chosen keys beat coverage.
• Skip correlated columns. Clustering on both order_date and ship_date wastes a key slot; they move together.
• Lead with the highest-selectivity, most-filtered column, and let join keys earn a slot only if they appear in filters too.

The Cosmos Thrace Perspective

We've delivered dozens of data platform implementations across Europe, many on Databricks, and the partition-or-cluster call is one we now make on almost every engagement. Three patterns keep repeating.

First, the tables teams want to convert first are rarely the ones that should move first. The instinct is to start with the biggest table; the better start is the table with the worst small-file profile and the most volatile query patterns, because that is where clustering pays back within days.

Second, the storage bump surprises everyone. The conversion itself is easy to sell; the temporary near-doubling of storage during re-clustering, on a table that is already the biggest line item in the workspace, is the part that needed a plan and usually did not have one.

Third, nobody regrets the flexibility. Every team we have migrated has changed at least one clustering key within the first months, something that would have been a full rewrite under partitioning. That, more than any benchmark, is the argument: liquid clustering turns your biggest irreversible design decision into a reversible one.

Where does this leave your tables?

Date partitioning earned its place as a habit, and habits outlive their reasons. For most Delta tables in 2026, liquid clustering is the better layout: faster on selective queries, immune to the small-files spiral, and, decisively, changeable when you get the keys wrong. The migration no longer requires a full rewrite, the automatic mode is usable within known limits, and the exceptions where partitioning still wins are few and specific.

The practical move is not "convert everything". It is: profile your estate, rank tables by layout pain, convert where scans hurt, verify pruning, and budget the storage window. Layout is one of the few levers that improves performance and cost at the same time. It deserves an afternoon of deliberate decision-making per table, not a reflex either way.

Weighing up a conversion across a real estate? Talk to our team or start with the free Databricks assessment.

Sources

• Databricks documentation: Use liquid clustering for Delta tables
• Azure Databricks documentation: Use liquid clustering for tables
• Databricks blog: Debunking 8 data layout myths (June 2026)
• Databricks Community discussions on liquid clustering migration and key selection
• dbt-databricks GitHub issue #1329 (streaming tables and liquid clustering)
• r/databricks community discussion on converting month-partitioned ingestion tables