Confirm Whether the Problem Is File Size or Data Layout
Small files are usually the result of a writing pattern rather than a broken component. Frequent streaming commits, short micro-batches, high-cardinality partitions, and concurrent producers can each create commits containing little data. The query engine must then list more objects, read more file footers, and schedule more scan tasks. Planning can become slow even when a query reads relatively little data.
Do not trigger compaction from average file size alone. Inspect the size distribution, files per partition, planning time, scanned bytes, and whether updates concentrate on the same partitions. A highly selective query may benefit from smaller files, while aggressive compaction can make it scan broader ranges. The right target also depends on interactive versus batch workloads, object-store latency, executor memory, compression, and the engine's ability to split large files.
Publish a New Snapshot Instead of Replacing Files in Place
Objects such as Parquet and ORC files should normally be treated as immutable. A safe compactor reads a committed set of small files, writes replacement files, validates them, and publishes a new table snapshot through a metadata transaction. Apache Iceberg, Delta Lake, and Apache Hudi implement this differently, but the central rule is the same: running queries retain their original version, and new queries see replacement files only after a successful commit. An incomplete rewrite must remain invisible.
A lake exposed as plain Parquet paths needs extra care. Never delete the source files before publishing their replacements, and do not treat a refreshed directory listing as a transaction. Write a complete result under a versioned prefix, validate it, and then switch a catalog entry, view, or manifest pointer. Keep the previous version available until rollback is no longer required.
- Freeze the input set:Record the starting snapshot or explicit file list so later arrivals are not included twice.
- Bound each rewrite:Operate on a partition, event-time range, or explicit file group rather than rewriting the whole table.
- Publish atomically:Finish data writes and validation before one metadata commit makes the result visible.
- Delay deletion:Retain old files beyond the longest query, time-travel requirement, and streaming-recovery window.
Coordinate with Streaming Writes, Updates, and Deletes
The hardest conflict is not usually a reader; it is another writer touching the same files. If MERGE, UPDATE, DELETE, or a streaming upsert overlaps a compaction batch, a table using optimistic concurrency may reject one commit. That rejection protects correctness. The compactor should refresh the current snapshot, re-plan the affected files, and retry a limited number of times instead of ignoring the conflict or forcing an overwrite.
Prefer compacting cold partitions that have passed the event-time watermark and rarely change. For hot partitions, raise the trigger threshold, reduce the batch scope, or use the table format's native optimize, clustering, or compaction operation. If historical corrections are valid business events, do not assume that old partitions are permanently read-only; detect active updates and skip those ranges for the current run.
Deletes also matter. Some formats represent them through position deletes, equality deletes, or change records rather than immediately rewriting data files. Verify whether compaction applies and absorbs those deletes. A custom rewrite that ignores them can restore logically deleted records. Data-file rewriting, delete-file consolidation, metadata cleanup, snapshot expiration, and orphan-file removal should be separate maintenance stages with explicit ordering.
Roll Out a Verifiable and Stoppable Maintenance Process
Start with a canary on a low-risk partition. Before committing, compare row counts, key uniqueness, required-column ranges, partition values, and file-level statistics; afterward, run representative correctness and latency queries. Equal row counts alone are not proof because duplicate and missing records can cancel each other out. Validation logs should also avoid exposing raw personal or sensitive values.
Schedule work from observed pressure, such as excessive files in a partition, an unhealthy size distribution, or material planning overhead, rather than rewriting every table on a fixed daily cycle. Monitor input and output file counts, rewritten bytes, commit conflicts, retries, orphaned objects, and query behavior. When conflicts or latency rise, stop subsequent batches before the affected scope grows.
- Expand gradually:Validate one partition or bounded file group before increasing concurrency and coverage.
- Preserve rollback time:Expire the old snapshot only after long queries and streaming checkpoints have advanced safely.
- Separate maintenance jobs:Give compaction, sorting, snapshot expiration, and orphan cleanup independent permissions, metrics, and failure handling.
- Retune the policy:Revisit thresholds as query patterns, ingestion frequency, storage behavior, and update semantics change.