execution.checkpointing.local-backup.enabled |
false |
Boolean |
This option configures local backup for the state backend, which indicates whether to make backup checkpoint on local disk. If not configured, fallback to execution.state-recovery.from-local. By default, local backup is deactivated. Local backup currently only covers keyed state backends (including both the EmbeddedRocksDBStateBackend and the HashMapStateBackend). |
state.backend.incremental |
false |
Boolean |
Option whether the state backend should create incremental checkpoints, if possible. For an incremental checkpoint, only a diff from the previous checkpoint is stored, rather than the complete checkpoint state. Once enabled, the state size shown in web UI or fetched from rest API only represents the delta checkpoint size instead of full checkpoint size. Some state backends may not support incremental checkpoints and ignore this option. |
state.checkpoint-storage |
(none) |
String |
The checkpoint storage implementation to be used to checkpoint state. The implementation can be specified either via their shortcut name, or via the class name of a CheckpointStorageFactory . If a factory is specified it is instantiated via its zero argument constructor and its CheckpointStorageFactory#createFromConfig(ReadableConfig, ClassLoader) method is called. Recognized shortcut names are 'jobmanager' and 'filesystem'. 'state.checkpoint-storage' and 'state.checkpoints.dir' are usually combined to configure the checkpoint location. By default, the checkpoint meta data and actual program state will be stored in the JobManager's memory directly. When 'state.checkpoint-storage' is set to 'jobmanager', if 'state.checkpoints.dir' is configured, the meta data of checkpoints will be persisted to the path specified by 'state.checkpoints.dir'. Otherwise, the meta data will be stored in the JobManager's memory. When 'state.checkpoint-storage' is set to 'filesystem', a valid path must be configured to 'state.checkpoints.dir', and the checkpoint meta data and actual program state will both be persisted to the path. |
state.checkpoint.cleaner.parallel-mode |
true |
Boolean |
Option whether to discard a checkpoint's states in parallel using the ExecutorService passed into the cleaner |
state.checkpoints.create-subdir |
true |
Boolean |
Whether to create sub-directories named by job id under the 'state.checkpoints.dir ' to store the data files and meta data of checkpoints. The default value is true to enable user could run several jobs with the same checkpoint directory at the same time. If this value is set to false, pay attention not to run several jobs with the same directory simultaneously. WARNING: This is an advanced configuration. If set to false, users must ensure that no multiple jobs are run with the same checkpoint directory, and that no files exist other than those necessary for the restoration of the current job when starting a new job. |
state.checkpoints.dir |
(none) |
String |
The default directory used for storing the data files and meta data of checkpoints in a Flink supported filesystem. The storage path must be accessible from all participating processes/nodes(i.e. all TaskManagers and JobManagers). If the 'state.checkpoint-storage' is set to 'jobmanager', only the meta data of checkpoints will be stored in this directory. |
state.checkpoints.file-merging.across-checkpoint-boundary |
false |
Boolean |
Only relevant if state.checkpoints.file-merging.enabled is enabled. Whether to allow merging data of multiple checkpoints into one physical file. If this option is set to false, only merge files within checkpoint boundaries. Otherwise, it is possible for the logical files of different checkpoints to share the same physical file. |
state.checkpoints.file-merging.enabled |
false |
Boolean |
Whether to enable merging multiple checkpoint files into one, which will greatly reduce the number of small checkpoint files. This is an experimental feature under evaluation, make sure you're aware of the possible effects of enabling it. |
state.checkpoints.file-merging.max-file-size |
32 mb |
MemorySize |
Max size of a physical file for merged checkpoints. |
state.checkpoints.file-merging.pool-blocking |
false |
Boolean |
Whether to use Blocking or Non-Blocking pool for merging physical files. A Non-Blocking pool will always provide usable physical file without blocking. It may create many physical files if poll file frequently. When poll a small file from a Blocking pool, it may be blocked until the file is returned. |
state.checkpoints.num-retained |
1 |
Integer |
The maximum number of completed checkpoints to retain. |
state.savepoints.dir |
(none) |
String |
The default directory for savepoints. Used by the state backends that write savepoints to file systems (HashMapStateBackend, EmbeddedRocksDBStateBackend). |
state.storage.fs.memory-threshold |
20 kb |
MemorySize |
The minimum size of state data files. All state chunks smaller than that are stored inline in the root checkpoint metadata file. The max memory threshold for this configuration is 1MB. |
state.storage.fs.write-buffer-size |
4096 |
Integer |
The default size of the write buffer for the checkpoint streams that write to file systems. The actual write buffer size is determined to be the maximum of the value of this option and option 'state.storage.fs.memory-threshold'. |
taskmanager.state.local.root-dirs |
(none) |
String |
The config parameter defining the root directories for storing file-based state for local recovery. Local recovery currently only covers keyed state backends. If not configured it will default to <WORKING_DIR>/localState. The <WORKING_DIR> can be configured via process.taskmanager.working-dir |