Configuration Options

We will all configuration options for Pulsejet here.

Pulsejet mainly configurable with config.toml file.

Before proceeding, if you feel alien to terminology, please check the terminology, design & concepts.

Example config toml is below:

rest_api_port = 47044

grpc_api_port = 47045



[storage]

prod_name = "pulsejetdb"

shard_id = 0

basedir = ".database"



[storage.hot_config]

hot_storage_size = 10000000

bloom_fp_rate = 0.001

bloom_size = 1000000

flush_size = 52428800

bufferpool_to_resident_ratio = 0.2

resident_writeback_interval_ms = 50000

txn_timeout_ms = 500



[storage.warm_config]

max_background_jobs = 10

wal_check_checksum = true

wal_checkpoint_segments = 3

wal_checkpoint_timeout_seconds = 300

wal_log_gc_seconds = 3600

wal_log_gc_percentage = 0.2



[storage.cold_config]

cold_enable = false

writeback_frequency_secs = 300

concurrent_requests = 100

bucket_name = "pulsejet_cold"

cloud_api = "GCP"

base_path = "snapshots"

Let’s explain the configuration options.

​

Ports

There are two APIs, one is HTTP the other is GRPC. GRPC used for performance queries.

rest_api_port = 47044

grpc_api_port = 47045

​

Storage

This section configures the storage options of this server.

[storage]

prod_name = "pulsejetdb"

shard_id = 0

basedir = ".database"

• prod_name: Indicates the name of this instance. If you want to give a naming or advertise the instance with a name you can change this.
• shard_id: This is both shard and node id. For single node cluster it can stay as 0. But for other instance you need to change it to identify the shards.
  • If shard id is left zero in cluster mode, cluster will automatically recon and name the nodes in consistent hashing method.
• basedir: This is the directory where all data will be stored. By default it is local directory of .database/.

​

Hot Storage

Pulsejet consists of hot, warm and cold storage tiers. This config is for hot storage (in-memory) config.

[storage.hot_config]

hot_storage_size = 10000000

bloom_fp_rate = 0.001

bloom_size = 1000000

flush_size = 52428800

bufferpool_to_resident_ratio = 0.2

resident_writeback_interval_ms = 50000

txn_timeout_ms = 500

• hot_storage_size: The amount of in-memory embeds that can be left in memory.
• bloom_fp_rate: False-positive rate of the bloom-filter to be used to find related embeds.
• bloom_size: Bloom filter size in general.
• flush_size: Flush size of the hot to warm storage. Keeping it bigger than hot_storage_size will make it flush to disk in a single shot.
• bufferpool_to_resident_ratio: How many embeds can be in the pre-buffer before goes into hot storage. For the example above it is 10000000 * 0.2.
• resident_writeback_interval_ms: How frequently hot storage will be committed to warm (on-disk).
• txn_timeout_ms: Transactional commit timeout in milliseconds.

​

Warm Storage

This is local mounted disk storage and it’s configuration.

[storage.warm_config]

max_background_jobs = 10

wal_check_checksum = true

wal_checkpoint_segments = 3

wal_checkpoint_timeout_seconds = 300

wal_log_gc_seconds = 3600

wal_log_gc_percentage = 0.2

• max_background_jobs: How many background jobs to be used to writeback to disk.
• wal_check_checksum: For WAL do we need to check the checksum on load.
• wal_checkpoint_segments: Segment size per WAL log block.
• wal_checkpoint_timeout_seconds: How frequently checkpointing should be made.
• wal_log_gc_seconds: How frequently WAL GC needs to run.
• wal_log_gc_percentage: Percentage of the GCed and resident WAL records. How much of them should be GCed is configurable by this.

​

Cold Storage

This is long term archival storage system. It is for writing data to object storages.

[storage.cold_config]

cold_enable = false

writeback_frequency_secs = 300

concurrent_requests = 100

bucket_name = "pulsejet_cold"

cloud_api = "GCP"

base_path = "snapshots"

• cold_enable: Enable cold storage, this is the switch for it.
• writeback_frequency_secs: How frequently we should write to object storage periodically.
• concurrent_requests: How many concurrent write operation can go simultaneously.
• bucket_name: Name of the bucket, bucket terminology can differ from one provider to another but the core model remains same.
• cloud_api: Which cloud API to use. Possible options are:
  • GCP for Google Cloud Storage.
  • AWS for AWS S3.
  • Azure for Azure Storage Blob.
• base_path: The path that will be following the bucket_name that we will write the data into.