Manifest#

Manifest List#

├── manifest
    └── manifest-list-51c16f7b-421c-4bc0-80a0-17677f343358-1

Manifest List includes metadata of several manifest files. Its name contains a UUID. It is an Avro file with the following schema:

  1. _FILE_NAME: STRING — manifest file name.

  2. _FILE_SIZE: BIGINT — manifest file size.

  3. _NUM_ADDED_FILES: BIGINT — number of added files in the manifest.

  4. _NUM_DELETED_FILES: BIGINT — number of deleted files in the manifest.

  5. _PARTITION_STATS: SimpleStats — partition stats. The minimum and maximum values of partition fields in this manifest are beneficial for skipping certain manifest files during queries.

  6. _SCHEMA_ID: BIGINT — schema id used when writing this manifest file.

Manifest#

Manifest includes metadata of several data files, changelog files, or table-index files. Its name contains a UUID, and it is an Avro file.

The changes of the file are saved in the manifest, and a file can be added or deleted. Manifests should be in an orderly manner, and the same file may be added or deleted multiple times. The last version should be read. This design makes commit lighter to support file deletion generated by compaction.

Data Manifest#

Data Manifest includes metadata of several data files or changelog files.

├── manifest
    └── manifest-6758823b-2010-4d06-aef0-3b1b597723d6-0

Schema:

  1. _KIND: TINYINT — ADD or DELETE.

  2. _PARTITION: BYTES — partition spec, a BinaryRow.

  3. _BUCKET: INT — bucket of this file.

  4. _TOTAL_BUCKETS: INT — total buckets when writing this file; used for verification after bucket changes.

  5. _FILE: data file metadata.

Data file metadata:

  1. _FILE_NAME: STRING — file name.

  2. _FILE_SIZE: BIGINT — file size.

  3. _ROW_COUNT: BIGINT — total number of rows (including add & delete) in this file.

  4. _MIN_KEY: STRING — minimum key of this file.

  5. _MAX_KEY: STRING — maximum key of this file.

  6. _KEY_STATS: SimpleStats — statistics of the key.

  7. _VALUE_STATS: SimpleStats — statistics of the value.

  8. _MIN_SEQUENCE_NUMBER: BIGINT — minimum sequence number.

  9. _MAX_SEQUENCE_NUMBER: BIGINT — maximum sequence number.

  10. _SCHEMA_ID: BIGINT — schema id when writing this file.

  11. _LEVEL: INT — level of this file in LSM.

  12. _EXTRA_FILES: ARRAY<STRING> — extra files for this file (e.g., data file index file).

  13. _CREATION_TIME: TIMESTAMP_MILLIS — creation time of this file.

  14. _DELETE_ROW_COUNT: BIGINT — rowCount = addRowCount + deleteRowCount.

  15. _EMBEDDED_FILE_INDEX: BYTES — if the data file index is small, store the index in the manifest.

  16. _FILE_SOURCE: TINYINT — indicates whether this file is generated as an APPEND or COMPACT file.

  17. _VALUE_STATS_COLS: ARRAY<STRING> — statistical columns in metadata.

  18. _EXTERNAL_PATH: STRING — external path of this file; null if it is in the warehouse.

Index Manifest#

Index Manifest includes metadata of several table-index files.

├── manifest
    └── index-manifest-5d670043-da25-4265-9a26-e31affc98039-0

Schema:

  1. _KIND: TINYINT — ADD or DELETE.

  2. _PARTITION: BYTES — partition spec, a BinaryRow.

  3. _BUCKET: INT — bucket of this file.

  4. _INDEX_TYPE: STRING — HASH or DELETION_VECTORS.

  5. _FILE_NAME: STRING — file name.

  6. _FILE_SIZE: BIGINT — file size.

  7. _ROW_COUNT: BIGINT — total number of rows.

  8. _DELETIONS_VECTORS_RANGES: Metadata only used by DELETION_VECTORS; an array of deletion vector metadata. Each deletion vector metadata has: - f0: the data file name corresponding to this deletion vector. - f1: the starting offset of this deletion vector in the index file. - f2: the length of this deletion vector in the index file. - _CARDINALITY: the number of deleted rows.

Appendix#

SimpleStats#

SimpleStats is a nested row with the following schema:

  1. _MIN_VALUES: BYTES — BinaryRow; the minimum values of the columns.

  2. _MAX_VALUES: BYTES — BinaryRow; the maximum values of the columns.

  3. _NULL_COUNTS: ARRAY<BIGINT> — the number of nulls in the columns.

BinaryRow#

BinaryRow is backed by bytes instead of Object. It can significantly reduce the serialization/deserialization of Java objects.

A row has two parts: fixed-length part and variable-length part.

  • Fixed-length part contains a 1-byte header, null bit set, and field values. The null bit set is used for null tracking and is aligned to 8-byte word boundaries.

  • Field values hold fixed-length primitive types and variable-length values that can be stored in 8 bytes. If the variable-length field does not fit in 8 bytes, then the fixed-length part stores the length and the offset of the variable-length part.