Write#

Interface#

class FileStoreWrite#

Interface for write operations in a file store.

Public Functions

virtual ~FileStoreWrite() = default#

virtual Status Write(std::unique_ptr<RecordBatch> &&batch) = 0#: Support write an input RecordBatch to internal buffer or file.

virtual Result<std::vector<std::shared_ptr<CommitMessage>>> PrepareCommit(bool wait_compaction = true, int64_t commit_identifier = BATCH_WRITE_COMMIT_IDENTIFIER) = 0#

Generate a list of commit messages with the latest generated data file meta information of the current snapshot.

When we need commit, call PrepareCommit to get the current CommitMessages with the latest generated data file meta information of the current snapshot.

This function is designed to be called when a commit is required. Depending on the writing scenario, the behavior will differ:

For batch write, simply call PrepareCommit() without any parameters.
For streaming write, you need to provide both parameters: PrepareCommit(bool wait_compaction, int64_t commit_identifier).

Parameters:

wait_compaction – Indicates whether to wait for any ongoing compaction process to complete.
commit_identifier – A unique identifier for the commit operation. This parameter is only relevant in streaming write scenarios.

Returns:

A Result containing std::vector<std::shared_ptr<CommitMessage>> objects, representing the generated commit messages.

virtual std::shared_ptr<Metrics> GetMetrics() const = 0#

virtual Status Close() = 0#

Public Static Functions

static Result<std::unique_ptr<FileStoreWrite>> Create(std::unique_ptr<WriteContext> context)#

Create an instance of FileStoreWrite.

Parameters:: context – A unique pointer to the WriteContext used for write operations.
Returns:: A Result containing a unique pointer to the FileStoreWrite instance.

class WriteContextBuilder#

WriteContextBuilder used to build a WriteContext, has input validation.

Public Functions

WriteContextBuilder(const std::string &root_path, const std::string &commit_user)#

Constructs a WriteContextBuilder with required parameters.

Parameters:

root_path – The root path of the table.
commit_user – The user identifier for commit operations.

~WriteContextBuilder()#

WriteContextBuilder &SetOptions(const std::map<std::string, std::string> &options)#

Set a configuration options map to set some option entries which are not defined in the table schema or whose values you want to overwrite.

Note

The options map will clear the options added by AddOption() before.

Parameters:: options – The configuration options map.
Returns:: Reference to this builder for method chaining.

WriteContextBuilder &AddOption(const std::string &key, const std::string &value)#

Add a single configuration option which is not defined in the table schema or whose value you want to overwrite.

If you want to add multiple options, call AddOption() multiple times or use SetOptions() instead.

Parameters:

key – The option key.
value – The option value.

Returns:

Reference to this builder for method chaining.

WriteContextBuilder &WithStreamingMode(bool is_streaming_mode)#

Set whether to enable streaming mode (default is false)

Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithIgnorePreviousFiles(bool ignore_previous_files)#

Set whether the write operation should ignore previously stored files.

(default is false)

Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithMemoryPool(const std::shared_ptr<MemoryPool> &memory_pool)#

Set custom memory pool for memory management.

Parameters:: memory_pool – The memory pool to use.
Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithExecutor(const std::shared_ptr<Executor> &executor)#

Set custom executor for task execution.

Parameters:: executor – The executor to use.
Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithWriteId(int32_t write_id)#

For postpone bucket mode in pk table, WithWriteId() supposed to be used.

Each worker must have its own unique write_id within a task, which is used as the prefix for its data files. This ensures that files from the same worker share the same prefix and can be consumed by the same compaction reader to preserve input order.

Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithBranch(const std::string &branch)#

Write to specific branch, default is main.

Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithWriteSchema(const std::vector<std::string> &write_schema)#

For data evolution, user can write partial specific fields from table schema.

If not set, write all fields in table.

Returns:: Reference to this builder for method chaining.

WriteContextBuilder &WithFileSystemSchemeToIdentifierMap(const std::map<std::string, std::string> &fs_scheme_to_identifier_map)#

Set the file system scheme to identifier mapping for custom file system configurations.

This allows using different file system implementations for different URI schemes.

Note

If not set, use default file system (configured in Options::FILE_SYSTEM).

Parameters:: fs_scheme_to_identifier_map – Map from URI scheme to file system identifier.
Returns:: Reference to this builder for method chaining.

Result<std::unique_ptr<WriteContext>> Finish()#

Build and return a WriteContext instance with input validation.

Returns:: Result containing the constructed WriteContext or an error status.

class WriteContext#

WriteContext is some configuration for write operations.

Please do not use this class directly, use WriteContextBuilder to build a WriteContext which has input validation.