IsleDB / Docs / Metrics reference

Metrics reference

Every IsleDB service serves Prometheus metrics at /metrics on its admin address, without authentication. This page lists them all, by service, with what to alert on. Each service also exports the standard Go process metrics.

Counters end in _total. Durations are in seconds, and times are Unix timestamps in seconds. Every service with an API reports its requests in the same two metrics:

Metric Shows
isledb_<service>_http_requests_total{route,code} Requests, by endpoint and status code.
isledb_<service>_http_request_seconds{route} How long requests took to answer, by endpoint.

Writer

Metric Shows
isledb_writer_status{status} 1 for the writer's current state: open, stopped, closed or fenced (another writer took over).
isledb_writer_epoch This writer's takeover count: each writer that takes over the database has a higher one.
isledb_writer_puts_total, isledb_writer_deletes_total Writes and deletes accepted.
isledb_writer_backpressure_total Writes refused because too much data was waiting to be saved.
isledb_writer_accepted_sequence The sequence number of the newest write accepted, saved or not.
isledb_writer_committed_sequence The sequence number of the newest write saved to storage.
isledb_writer_pending_bytes, isledb_writer_pending_memtables Data accepted but not yet saved.
isledb_writer_oldest_uncommitted_timestamp_seconds When the oldest write still waiting to be saved arrived; 0 when nothing is waiting.
isledb_writer_commits_total, isledb_writer_commit_errors_total Saves to storage, and saves that failed and are being retried.
isledb_writer_commit_seconds How long a save takes.
isledb_writer_commit_bytes_total Bytes saved to storage.
isledb_writer_http_inflight_body_bytes Size of the requests being read and applied right now.
isledb_writer_l0_merges_total, isledb_writer_l0_merge_errors_total The writer combines its recent saves into larger files: combined files uploaded, and attempts that failed and are retried.
isledb_writer_l0_merge_bytes_total Bytes of combined files uploaded.
isledb_writer_l0_merge_held_bytes Recent saves held in memory to combine.
isledb_writer_l0_merge_skipped_bytes_total Saves not held for combining because the memory for it was full.
isledb_writer_l0_merges_published_total, isledb_writer_l0_merges_refused_total Combined files that replaced the saves they came from, and those refused because maintenance had already changed those saves.
isledb_writer_l0_merges_dropped_total Combining work given up because maintenance took the files first.
isledb_writer_maintenance_publish_delay_seconds How long finished maintenance work waited before the writer published it, from when maintenance handed it over. Usually about a second.
isledb_writer_maintenance_apply_seconds{outcome} How long the writer took to publish one piece of maintenance work, by outcome: applied, rejected (refused; maintenance redoes it) or failed (tried again).

Alert on:

Reader

Metric Shows
isledb_reader_view_position The revision reads are answered from. Compare it across replicas, or with the writer's commits, to see how far behind a replica is.
isledb_reader_view_loaded_timestamp_seconds When the reader last loaded new data; time() minus it is how old the data being served is.
isledb_reader_view_outdated_since_timestamp_seconds For a reader that publishes new data only when told to: since when it has held back newer data; 0 otherwise.
isledb_reader_view_safety_publishes_total Times such a reader published newer data itself, after holding it back too long.
isledb_reader_refresh_total, isledb_reader_refresh_errors_total Loads of new data, and loads that failed.
isledb_reader_refresh_latency_seconds How long a load of new data takes.
isledb_reader_stale_reads_total Reads answered while loading new data was failing.
isledb_reader_get_total, isledb_reader_get_hits_total, isledb_reader_get_misses_total, isledb_reader_get_errors_total Key reads: all, found, not found, and failed.
isledb_reader_get_latency_seconds How long a key read takes.
isledb_reader_scan_total, isledb_reader_scan_errors_total, isledb_reader_scan_results_total, isledb_reader_scan_latency_seconds Scans: count, failures, keys returned, and duration.
isledb_reader_scan_limit_total, isledb_reader_scan_limit_errors_total, isledb_reader_scan_limit_results_total, isledb_reader_scan_limit_latency_seconds Scans with a limit, such as one page of GET /v1/kv: the same four.
isledb_reader_http_reads_in_progress Reads being served now.
isledb_reader_sst_range_read_total, isledb_reader_sst_range_read_errors_total Reads of data files from storage, and those that failed.
isledb_reader_sst_range_read_bytes_total, isledb_reader_sst_range_read_latency_seconds Bytes those reads fetched, and how long they took.
isledb_reader_bloom_filter_skips_total Key reads that checked a file without its filter, which lets a read skip files that can't hold the key, because the filter was still loading.
isledb_reader_bloom_filter_errors_total Filters that failed to load; reads go on without them.
isledb_reader_dead_ssts_pending Replaced data files not yet removed from the disk cache.
isledb_reader_read_cache_reads_total Reads the read cache answered.
isledb_reader_read_cache_fallbacks_total{reason} Reads meant for the read cache that went to storage instead, and why.
isledb_reader_read_cache_doorbells_total{result} Waits on the read cache for new writes, by result: rang (new data), quiet or error.
isledb_reader_read_cache_current_total{outcome} Loads of new data asked of the read cache, by outcome: relayed, too_old, behind or failed.
isledb_reader_read_cache_current_age_seconds How old the read cache's copy of the newest version was, when last asked.

Alert on:

Maintenance

Metric Shows
isledb_maintenance_running 1 while maintenance works on the database; 0 after another maintenance process took it over.
isledb_maintenance_cycles_total, isledb_maintenance_cycle_errors_total Rounds of work, and rounds that failed and are retried.
isledb_maintenance_last_cycle_timestamp_seconds When the last round completed. A round lasts as long as its compaction, which can be many minutes for a large one.
isledb_maintenance_last_progress_timestamp_seconds When maintenance last made progress: it opened, finished a round, or, within a compaction, copied an input file or saved an output file. A compaction does one of those every few seconds, however large it is.
isledb_maintenance_waiting_for_writer_since_timestamp_seconds When the work the writer has yet to publish was handed to it; 0 when nothing waits. Maintenance sees the writer has published when its next round starts, and updates this then.
isledb_maintenance_compaction_jobs_total Compactions, counted when they run, including any the writer then refuses.
isledb_maintenance_compaction_l0_merges_total, isledb_maintenance_compaction_moves_total Of those, merges of newly saved files with each other, and moves of files to another level without rewriting them.
isledb_maintenance_compaction_read_bytes_total, isledb_maintenance_compaction_rewritten_bytes_total Bytes compaction read from and wrote to storage: most of maintenance's storage cost.
isledb_maintenance_compaction_moved_bytes_total Bytes moved without being rewritten.
isledb_maintenance_compaction_oversized_jobs_total Compactions larger than usual, run because the level they write into could not be made smaller first. Each writes one unusually large file.
isledb_maintenance_compaction_seconds{kind} How long compactions take, by kind: l0_drain (new files into the first level), l0_merge (new files with each other), level (one level into the next) and move (files moved without being rewritten).
isledb_maintenance_compaction_jobs_by_level_total{level} Compactions, by the level they read from: 0 for the newest files, 1 for the first level, and so on. A level whose level_bytes stays above its target while its count stands still is not getting turns.
isledb_maintenance_compaction_blocked_total{level} Times no job could be planned for a level due for compaction, by level. Maintenance tries again on its next round. A level waiting its turn is not counted.
isledb_maintenance_compaction_critical 1 when the last compaction was urgent, because the newest files had piled up too deep.
isledb_maintenance_l0_files, isledb_maintenance_l0_bytes Newly saved data not yet merged into the first level. This and the level metrics below are read from the database every 10 seconds, also while a long compaction runs.
isledb_maintenance_l0_depth The most of those files one key is in, so the most a read of one key may check. At compaction.l0_max_depth, they are merged into the first level; past it, compaction is behind.
isledb_maintenance_level_files{level}, isledb_maintenance_level_bytes{level} Files and bytes in each level below the newest files.
isledb_maintenance_level_target_bytes{level} The size compaction keeps each level near. A level far above it makes every merge into it more expensive.
isledb_maintenance_shape_timestamp_seconds When the L0 and level metrics were last read. If it falls behind, storage is failing; the metrics keep their last values.
isledb_maintenance_commands_rejected_total Finished work the writer refused to publish. The writer logs why.
isledb_maintenance_compaction_rejected_bytes_total Bytes compaction wrote for work the writer refused. The files are deleted later.
isledb_maintenance_checkpoints_total Checkpoints of the database's metadata prepared for the writer to publish.
isledb_maintenance_deleted_objects_total{kind}, isledb_maintenance_delete_failures_total{kind} Objects deleted from storage, and deletes that failed and are retried, by kind: sst, change_batch, snapshot, page.

Alert on:

Read cache

Metric Shows
isledb_cache_chunks_total{source} Pieces of data served, by where they came from: disk, storage (read for this request), shared (read for another request at the same time) or bridged (on disk, read again to fill a gap).
isledb_cache_object_responses_total{status} Responses, by the Isledb-Cache header: hit, shared or miss.
isledb_cache_storage_reads_total, isledb_cache_storage_read_bytes_total Reads sent to storage, and the bytes they returned.
isledb_cache_storage_reads_queued_total Storage reads that waited for their turn (reads.max_bytes).
isledb_cache_storage_bytes_reading Bytes being read from storage now.
isledb_cache_size_lookups_total Object sizes asked of storage, for requests that didn't give one.
isledb_cache_current_position{db} The newest revision the cache has seen, by database.
isledb_cache_current_age_seconds{db} How long since the cache last checked a database for new writes.
isledb_cache_current_poll_errors_total{db} Checks for new writes that failed.
isledb_cache_long_polls_held Readers waiting for new writes now.
isledb_cache_disk_bytes, isledb_cache_disk_max_bytes, isledb_cache_disk_entries Data on disk, the disk budget, and the number of pieces held.
isledb_cache_disk_evictions_total, isledb_cache_disk_failures_total Pieces removed to make room, and pieces that couldn't be written to disk.

route on the cache's request metrics is current, current_long_poll, object or other.

Alert on: