Data Cleanup

Note

The entire data cleanup feature supports only append tables, and cleanup of index manifests, changelog, statistics are not supported. Do not use this feature with primary key tables.

This document describes three cleanup capabilities:

• Orphan File Cleanup

• Expiring Partitions

• Expiring Snapshots

Orphan File Cleanup

Description

• Orphan file cleanup currently supports only append tables.

• It runs as an independent task: construct a CleanContext and launch an OrphanFilesCleaner.

Detailed Steps

1. List all Paimon-specific subdirectories under the table directory (e.g., manifest/, snapshot/, f1=10/bucket-0, …).

2. Based on the subdirectories from step 1, enumerate all Paimon files in the table directory.

3. Using snapshot information, determine all in-use manifest files and data files.

4. Compute the set of files that appear in step 2 but not in step 3 (i.e., orphan files). Among those, delete files whose modification time is earlier than older_than_ms.

Performance Considerations

• Steps 1, 2, and 4 use an executor to parallelize I/O operations.

• Orphan file cleanup may take a long time; you can pass an executor with more threads to accelerate the process.

TODO

• Support cleanup of index manifests, changelog, statistics, etc.

Expiring Partitions

Description

• Executed within the Commit task. First construct a FileStoreCommit.

• DropPartition uses a mark-delete strategy. Calling DropPartition will send an Overwrite message, marking all data files in the specified partition as DELETE.

• A new snapshot is committed afterward. Actual deletion of data files occurs as snapshots expire.

Detailed Steps

1. Build a ScanFilter for the partition and use the latest snapshot to scan the partition.

2. Iterate over the scanned data file list (ManifestEntries) and rewrite each entry’s type to DELETE.

3. Commit using the rewritten ManifestEntries. If the commit fails, retry a limited number of times.

Expiring Snapshots

Description

• Executed within the Commit task. First construct a FileStoreCommit.

• The following optional configuration parameters control snapshot expiration:

  • snapshot.num-retained.min: The minimum number of completed snapshots to retain (>= 1). Default: 10.

  • snapshot.num-retained.max: The maximum number of completed snapshots to retain (>= snapshot.num-retained.min). Default: int32 max value.

  • snapshot.time-retained: The maximum age of completed snapshots to retain. Default: 1 hour.

  • snapshot.expire.limit: The maximum number of snapshots allowed to expire at a time. Default: 10.

• The snapshot expiration interface deletes data files according to the expiration policy and returns the number of snapshots deleted.

Detailed Steps

1. Use snapshot_manager to find earliest_snapshot_id and latest_snapshot_id.

2. Based on earliest/latest_snapshot_id and the expire config, determine the range of snapshots to clean.

Note

Consumer subscription management (consumer manager) is not currently supported. Users must ensure that snapshots to be expired are not in use.

3. Verify that the snapshot range is continuous. Normally, it is continuous. If a snapshot is missing, assume earlier snapshots were already cleaned and the missing files are orphaned remnants due to I/O exceptions; they are out of scope for this cleanup.

4. Clean data files for the updated expiration range.

   • To decide whether a file from a snapshot should be deleted, check if it was marked DELETE in the delta of the subsequent snapshot.

   • For an expiration range [begin, end), iterate over (begin,end] and delete data files whose type is DELETE in each snapshot.DeltaManifestList().

   • If a file underwent multiple ADD and DELETE operations, deletion follows the operation order:

     • ADD then DELETE → the file is deleted.

     • DELETE then ADD → the initial DELETE does not apply (file did not exist yet); the subsequent ADD ensures the file remains.

5. Clean meta files: - Preserve manifests used by the last snapshot in the cleanup range (end_exclusive_id). - Delete manifest files used by snapshots from begin_inclusive_id to end_exclusive_id (exclusive) and delete the snapshot files themselves.

6. Rewrite EarliestHint to end_exclusive_id.

7. Return the number of snapshots deleted, i.e., end_exclusive_id - begin_inclusive_id.

Performance Considerations

• Step 4 uses an executor to parallelize file deletions.

• If deletion is slow, pass an executor with more threads to accelerate the process.

TODO

• Preserve tag (savepoint) data via tagManager.

• Delete changelog files.

• Remove empty directories.