IsleDB / Docs / Read cache

Read cache

Run the cache

isledb cache is the read cache, run as a service. It serves one or more databases to the readers in a region. It checks each database for new writes every 500 ms and tells waiting readers at once, and it keeps the data readers ask for on its local disk, so the bucket is read once for all of them instead of once for each. Readers use it through the cache HTTP API.

isledb cache --config /etc/isledb/isledb.yaml

or, without a file:

isledb cache \
  --bucket-url 's3://my-bucket?region=us-east-1' \
  --db prod/policy --db prod/flags \
  --disk-dir /var/cache/isledb --disk-bytes 200GiB \
  --auth-token-file /etc/isledb/tokens

Storage credentials come from the provider's usual environment (for S3, the AWS SDK chain: environment variables, shared config, instance or pod role). They are never set in the cache's own configuration.

Configuration

Each setting can come from the YAML file, an environment variable or a flag. A flag overrides the environment, the environment overrides the file, and the file overrides the default. --db on the command line, or ISLEDB_CACHE_DATABASES, replaces the file's list of databases rather than adding to it.

One file can configure every IsleDB service: settings they share sit at the top, and each service reads its own section. The cache reads cache:.

bucket_url: s3://my-bucket?region=us-east-1
log_level: info

cache:
  listen: :8090
  admin_listen: 127.0.0.1:8091

  disk:
    dir: /var/cache/isledb
    bytes: 200GiB

  reads:
    max_bytes: 256MiB
    max_wait: 5s

  current:
    poll: 500ms
    long_poll: 30s

  databases:
    - prefix: prod/policy
    - prefix: prod/flags
      current_poll: 250ms

  auth:
    token_file: /etc/isledb/tokens

  tls:
    cert: /etc/isledb/tls/cert.pem
    key: /etc/isledb/tls/key.pem
    client_ca: /etc/isledb/tls/clients.pem

  shutdown_drain: 5s

A database at the root of the bucket is written /. An empty prefix is an error, so a blank value never names the root, and so is one with an empty, . or .. segment (prod//a, prod/../a), whose keys would never match.

isledb cache --config isledb.yaml --check validates the configuration, prints it as resolved from every source, and exits.

Every setting, with its flag, environment variable and default, is in the configuration reference.

Authentication

The cache refuses to start without authentication unless cache.auth.insecure is set. Readers send a token from the cache's auth.token_file, and with tls.client_ca also a client certificate; an isledb reader sets these with its read_cache settings. See Secure IsleDB.

Each database is served only its own objects: those right under its prefix, not those of a database nested under it (prod/a/b under prod/a), and, for the root, not those of the databases under prefixes. A client can read through the cache only the databases it is configured to serve.

Admin endpoints

On admin_listen, without authentication; keep it on an address only operators and scrapers reach.

Metrics

The cache's metrics, and what to watch, are in the metrics reference: above all, storage not answering, reads waiting for their turn, and a disk too small for what readers read.

Shutdown

On SIGTERM or SIGINT the cache:

1. reports not ready on /readyz;
2. waits cache.shutdown_drain for load balancers to notice;
3. answers every held long poll with the position it has;
4. finishes requests in flight, for up to 30 seconds, and exits.

A second SIGTERM or SIGINT during these steps ends the process at once.

Chunks on disk are kept. After a restart, requests carrying Isledb-Object-Size are served from them at once.

Deploying

• Disk. Put cache.disk.dir on local SSD. One process per directory: the chunk store locks it.
• Timeouts in front of the cache. Load balancers and proxies must allow idle connections longer than cache.current.long_poll, or they cut long polls off. An isledb reader waits up to 10 seconds for the cache, then reads storage itself. cache.reads.max_wait, 5 seconds by default, is about half that, so a read that waited for its turn still has time to fetch and answer. Raise it only for clients that wait longer, to about half their timeout.
• Several replicas. Round-robin works: each replica checks for new writes on its own, and readers treat a copy behind their view as nothing new. Each replica then caches every hot chunk. Hashing object requests by path at the load balancer (Envoy ring hash, NGINX hash $uri consistent) puts each chunk on one replica, so their disks add up instead of repeating each other. /v1/current can stay round-robin.
• Readiness. Route traffic by /readyz, not /healthz.

Kubernetes

The admin address defaults to 127.0.0.1:8091, which the kubelet's probes cannot reach from outside the container. Bind it to all interfaces in the pod, and keep it off the Service.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: isledb-cache
spec:
  serviceName: isledb-cache
  replicas: 2
  selector:
    matchLabels: { app: isledb-cache }
  template:
    metadata:
      labels: { app: isledb-cache }
    spec:
      # Longer than cache.shutdown_drain (5s) plus the 30s given to
      # requests in flight.
      terminationGracePeriodSeconds: 45
      securityContext:
        fsGroup: 65532   # the image's user: lets it write the chunk volume
      containers:
        - name: cache
          image: ghcr.io/ankur-anand/isledb:0.20.0
          args: ["cache", "--config", "/etc/isledb/isledb.yaml"]
          env:
            - name: ISLEDB_CACHE_ADMIN_LISTEN
              value: "0.0.0.0:8091"
          ports:
            - { name: api, containerPort: 8090 }
            - { name: admin, containerPort: 8091 }
          readinessProbe:
            httpGet: { path: /readyz, port: admin }
            periodSeconds: 2
          livenessProbe:
            httpGet: { path: /healthz, port: admin }
            periodSeconds: 10
          volumeMounts:
            - { name: config, mountPath: /etc/isledb, readOnly: true }
            - { name: chunks, mountPath: /var/cache/isledb }
      volumes:
        - name: config
          projected:
            sources:
              - configMap: { name: isledb-config }   # isledb.yaml
              - secret: { name: isledb-tokens }      # tokens
  volumeClaimTemplates:
    - metadata: { name: chunks }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: local-ssd
        resources: { requests: { storage: 200Gi } }
---
apiVersion: v1
kind: Service
metadata:
  name: isledb-cache
spec:
  selector: { app: isledb-cache }
  ports:
    - { name: api, port: 8090, targetPort: api }   # the admin port is not here
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isledb-cache-admin
spec:
  podSelector:
    matchLabels: { app: isledb-cache }
  policyTypes: [Ingress]
  ingress:
    - ports: [{ port: 8090 }]                 # the API, from anywhere it is routed
    - ports: [{ port: 8091 }]                 # the admin port, from monitoring only
      from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: monitoring }

• Why a StatefulSet. Each replica keeps its own chunk volume across restarts, so a restarted pod serves hits from disk at once.
• Probes. Readiness uses /readyz, so a pod takes traffic only once it has checked every database for new writes; liveness uses /healthz, which does not depend on storage, so a storage outage never restarts the pods. Kubelet probes come from the node, which most network plugins let through regardless of policy; if yours does not, allow the node's addresses on port 8091.
• Admin port. Not in the Service, and only the monitoring namespace may reach it: it serves /debug/pprof. Scrape /metrics through the pod's admin port.
• Shutdown. The cache drains on its own: on SIGTERM it reports not ready, waits cache.shutdown_drain, then answers held long polls. No preStop hook is needed.
• Mixed versions. A pod whose binary lacks a role named in a shared isledb.yaml refuses to start, so roll the image before adding a new role's section to the shared file.

Cache HTTP API

The read cache sits between IsleDB readers in one region and object storage. It serves two things:

• CURRENT, the one mutable object, relayed from a single copy the cache reads from storage every 500 ms.
• Immutable objects (SSTs, change batches, manifest pages and snapshots), by byte range, from 128 KiB chunks kept on the cache's disk.

Running it: Read cache.

Version: v1. Every path starts with /v1/.

Endpoints

P is a database's prefix in the bucket, as configured on the cache, and / for a database at the bucket root. db may be left out when the cache serves one database.

{key} is the object's full storage key, including the database prefix, exactly as the reader's store names it, for example db/sstable/a3f/000123.sst. Only objects of a database the cache serves are answered (see below).

Authentication

Every request to these endpoints carries Authorization: Bearer <token>, or comes over TLS with a client certificate, as the cache is configured. A request without valid credentials answers 401 with WWW-Authenticate: Bearer.

GET /v1/current

Returns the cache's copy of CURRENT, or, as a long poll, only its position once it passes a given one. A long poll is a doorbell: a reader that sees a new position reads CURRENT with a plain GET /v1/current, so CURRENT reaches readers by one path, where the client rules below apply.

Log position. The position is CURRENT's next_seq: the sequence number the next manifest log entry will get. Every entry appended to the manifest log raises it by one, in the same conditional write of CURRENT that adds the entry. Entries record changes to the database's structure: a flush adding an SST (with its change batch), a compaction, SSTs removed, a checkpoint, or a writer or compactor taking ownership. User writes do not move it until a flush. It never goes down. Some rewrites of CURRENT add no entry and keep the position, such as folding the log into a new snapshot.

Query

Response headers

Without after, the body is CURRENT's bytes as stored. With after, the body is empty.

Status

Behaviour

• The copy never moves back. A CURRENT in storage at a lower position is refused and logged; a store recreated under the same prefix needs the cache restarted.
• A CURRENT at the same position with other bytes, such as one rewritten around a new snapshot, replaces the copy. It does not answer long polls, which wait for a higher position; the next request gets the new bytes.
• When storage cannot be read, the cache keeps serving its copy, and its age grows. Readers decide whether it is still young enough.

Client rules (enforced by the reader, so a faulty cache cannot break it)

1. Age limit. Reject a CURRENT whose age is above 5 seconds and read storage directly.
2. Load time = the time the request was sent (own clock) − the age. This is when the view's MaxPinnedViewAge starts.
3. Older means read storage. A CURRENT from the cache at a position below the reader's published view, as a lagging replica can give, is not replayed: the refresh reads CURRENT from storage instead, which also renews the view.
4. Refresh reads storage directly. The relay can be up to one poll interval behind.
5. Long poll. Ask with after=<published position>; on an answer above it, call RequestRefresh and ask again. RequestRefresh reads CURRENT with a plain GET, where rules 1 to 3 apply. Give up on a long poll unanswered after 45 seconds: the connection is dead. An answer not above it whose age is past the limit of rule 1 means the cache cannot read storage and will not ring for new commits: read CURRENT, which then goes to storage. After a timeout or error, wait a random 0 to 1 second before asking again.
6. Fallback. On a connection error, a timeout or a 5xx, read storage directly.

Example

$ curl -si -H "Authorization: Bearer $TOKEN" 'http://cache:8090/v1/current?db=prod/policy&after=41'
HTTP/1.1 200 OK
Isledb-Age-Ms: 3
Isledb-Position: 42
Content-Length: 0

$ curl -si -H "Authorization: Bearer $TOKEN" 'http://cache:8090/v1/current?db=prod/policy'
HTTP/1.1 200 OK
Content-Type: application/json
Isledb-Age-Ms: 212
Isledb-Position: 42

{"layout_version":…,"next_seq":42,…}

GET /v1/object/{key}

Returns bytes of an immutable object.

Keys served. Only objects that are never rewritten, each a fixed shape right after its database's prefix P (nothing before the shape for the root):

A key is answered only when what comes before its shape is exactly a database the cache serves. So each database is served only its own objects: serving prod/a does not serve prod/a/b/sstable/… of a database nested under it, and serving the root does not serve prod/a/sstable/…. Anything else, including manifest/CURRENT and maintenance/HEAD, answers 403.

Request headers

Response headers

Status

A 404 is retried against storage because a cache pointed at the wrong bucket or prefix would answer 404 for everything; readers must not trust it alone.

The size

• The size must be the object's full stored length. For an SST with a Bloom sidecar that is BloomOffset + BloomLength + 16: the sidecar follows the SST's Size, and a 16-byte trailer follows the sidecar. Without one, it is Size. For a change batch it is the batch's Size.
• Chunks are kept per key and size. A request with a wrong size never reads or replaces chunks kept for the right one.
• Every read from storage checks the size against the size storage reports, at no extra request. A wrong size is refused (400) and nothing is kept, so only bytes read under the right size are ever served from disk.
• Without the header the cache asks storage once (a HEAD) and remembers the size in memory until restart.
• A wrong size larger than the object can instead fail at storage, which refuses a range past the end; the cache then answers 502.

Behaviour

• Objects are kept as 128 KiB chunks, aligned at offset 0, the same size as the reader's own chunks. The last chunk is shorter.
• Missing chunks next to each other are read from storage in one request, up to 32 chunks (4 MiB). A request also reads again up to 8 chunks it already has, to join two missing runs into one request.
• Concurrent requests for the same missing chunk share one storage read.
• A storage read is not tied to the request that started it: a client that disconnects does not fail others waiting on the same chunks.
• Storage reads in progress are capped in bytes (MaxReadBytes, default 256 MiB). Past the cap, a read waits its turn, in arrival order, for at most MaxReadWait (default 5 seconds), and then fails with 502. The wait does not shorten the storage request's own 30-second timeout. Set MaxReadWait to about half the readers' timeout to the cache (10 seconds by default), so a read that gets its turn still has time to fetch and answer: a queued read keeps its place after its reader gives up, so a longer wait spends budget on fetches whose requesters have left, ahead of requests still waiting. Hits are never queued.
• A response is sent 4 MiB at a time. The status and headers are sent once the first 4 MiB are in hand, so an error there gets its own status. A failure after that ends the body early; the client sees fewer bytes than Content-Length.
• The cache does not check object contents. Readers verify every object as they do when reading storage directly (SST, Bloom, page, snapshot and change-index checksums, and Pebble block checksums).

Examples

A lookup's chunk, cold and then warm:

$ curl -si -H "Authorization: Bearer $TOKEN" http://cache:8090/v1/object/db/sstable/a3f/000123.sst \
    -H 'Range: bytes=131072-262143' -H 'Isledb-Object-Size: 8421376'
HTTP/1.1 206 Partial Content
Content-Range: bytes 131072-262143/8421376
Content-Length: 131072
Isledb-Cache: miss
…

The next request for that chunk answers Isledb-Cache: hit.

A manifest page, whole:

$ curl -s -H "Authorization: Bearer $TOKEN" http://cache:8090/v1/object/db/manifest/pages/l00/p17.page.zst \
    -H 'Isledb-Object-Size: 20381' -o p17.page.zst

A scan that should not fill the cache:

$ curl -s -H "Authorization: Bearer $TOKEN" http://cache:8090/v1/object/db/sstable/a3f/000123.sst \
    -H 'Range: bytes=0-4194303' -H 'Isledb-Object-Size: 8421376' \
    -H 'Isledb-No-Cache: 1'

Reader support

Readers configured with a read cache read SSTs, manifest pages and snapshots through it, with fallback to storage, and refresh through the relay with the client rules above and the long poll. Change readers configured with a read cache read change batch indexes and blocks, the feed's view and its manifest pages through it, with the same rules and fallback, and wait for changes with the long poll.