IsleDB / Docs / HTTP API reference

HTTP API reference

The writer service takes writes; reader services serve reads, scans, and the change feed. Both speak JSON over HTTP, from any language.

Writer API

The writer service stores keys and values in one IsleDB database. You can write a key, delete a key, write several keys at once, and check whether a write was saved.

To run the service, see Writer service.

Every path starts with /v1/. Every response body is JSON.

Quick start

# Write a value
curl -X PUT -H "Authorization: Bearer $TOKEN" \
  --data '{"name":"a"}' 'http://isledb-writer:8070/v1/kv/user/42'
# {"write_id":"7.10442","revision":"1042"}

# Delete it
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  'http://isledb-writer:8070/v1/kv/user/42'

Authentication

Send your token in every request:

Authorization: Bearer <token>

Without a valid token, the writer answers 401.

Endpoints

Keys and values

• Keys go in the URL path. A key like user/42 can be sent as it is: /v1/kv/user/42. URL-encode a key (%XX) when it holds %, ?, #, a space or anything other than printable ASCII, or when it has an empty, . or .. part between slashes, like a//b or ../x: clients and proxies change those before the request arrives. Encoding every character that is not a letter, digit, -, _, . or ~ is always safe, so user/42 may also be written user%2F42. Any byte can be part of a key.
• A value can be any bytes, including nothing at all.
• In a batch, keys and values are text (key, value) or base64 bytes (key_base64, value_base64).

Write a key

PUT /v1/kv/user/42
Content-Type: application/json

{"name":"a"}

200 OK

{"write_id":"7.10442","revision":"1042"}

The writer answers once the value is saved, usually within a few hundred milliseconds. A 200 means the value is stored durably.

The response has two fields:

• write_id identifies this write. Use it to check on the write later, for example after a timeout.
• revision identifies the version of the database that includes your write. Keep it if you need to read your own write back: a read at this revision or later always includes it. Treat it as an opaque string; don't compare revisions or do arithmetic with them.

Expiring values. Add ?ttl= to make the value expire after a while:

PUT /v1/kv/session/abc?ttl=24h

Durations look like 500ms, 30s, 10m or 24h. Without ttl, the value never expires. An empty ttl= is an error, so a missing value in your code doesn't silently store a key forever.

Delete a key

DELETE /v1/kv/user/42

200 OK

{"write_id":"7.10443","revision":"1043"}

Deleting a key that doesn't exist is not an error.

Write several keys at once

Send a list of operations. They are applied together: anyone reading the database sees either all of them or none of them.

POST /v1/kv
Content-Type: application/json

{"ops":[
  {"op":"put","key":"order/42","value":"{\"total\":9}","ttl":"24h"},
  {"op":"put","key":"stock/7","value_base64":"MTI="},
  {"op":"delete","key":"cart/42"}
]}

200 OK

{"write_id":"7.10446","revision":"1044"}

• op is put or delete.
• Give the key as key, text, or as key_base64, base64 bytes; not both. Text is stored as its UTF-8 bytes, so for any other bytes use base64. A body that is not valid UTF-8 is refused with 400, and so is one with a lone \uD800–\uDFFF escape, half a surrogate pair, which is not text.
• A put's value works the same way: value or value_base64, not both. Without either, the value is empty.
• ttl is optional on a put.
• A delete has only a key.
• If the same key appears twice, the last operation wins.

Batches are much faster than one request per key when you write a lot of data. A batch can hold up to about 16 MiB of keys and values. If it's too large, the writer answers 413; split it into smaller batches.

Write without waiting

To get an answer as soon as the writer has accepted a write, without waiting for it to be saved, add this header:

Prefer: respond-async

202 Accepted
Location: /v1/writes/7.10447

{"write_id":"7.10447"}

The write is saved shortly afterwards. Until then it can still be lost, for example if the writer crashes. To find out when it's saved, check it at the Location URL.

Waiting less. To wait at most a few seconds for a write to be saved, send Prefer: wait=N, where N is a number of seconds. If the write isn't confirmed in time, the writer answers 504 with its write_id.

Check a write

GET /v1/writes/7.10442

200 OK

{"status":"committed","revision":"1042"}

A writer that was replaced, and can't tell what happened to its own write, answers 503 unavailable instead of unknown: the writer now holding the database can answer. Ask again after Retry-After seconds.

Sending a request body

PUT and POST requests must include a Content-Length header. Most HTTP clients add it automatically: curl with --data, Python requests, JavaScript fetch with a string body, and Go with a byte slice. Requests without it are answered 411.

Errors

Errors have a JSON body:

503 Service Unavailable
Retry-After: 1

{"error":"backpressure","message":"too many writes waiting to commit; retry"}

Retrying safely

• 503 with backpressure or unavailable: nothing was written, so retry the same request.
• 503 with not_committed: the write will never be saved, so send it again.
• 504: the write may or may not be saved. Check its status first. Sending it again without checking can apply it twice. That's harmless when you write the same value to the same key, but not for anything else.

Reader API

The reader service reads keys, and scans ranges of keys, from one IsleDB database. Run as many readers as you like: each serves reads on its own, so add replicas to serve more.

To run the service, see Reader service. Writes go to the writer service, described in the writer HTTP API.

Every path starts with /v1/.

Quick start

curl -H "Authorization: Bearer $TOKEN" 'http://isledb-reader:8080/v1/kv/user/42'
# {"name":"a"}

Authentication

Send your token in every request:

Authorization: Bearer <token>

Without a valid token, the reader answers 401.

Endpoints

The paths match the writer's, so a gateway can serve one API: send GET and HEAD requests for /v1/kv, /v1/kv/… and /v1/changes to readers, and everything else, including GET /v1/writes/…, to the writer.

Read a key

GET /v1/kv/user/42

200 OK
Content-Type: application/octet-stream
Content-Length: 12
Isledb-Revision: 1042

{"name":"a"}

• Keys go in the URL path, as for the writer. A key like user/42 can be sent as it is: /v1/kv/user/42. URL-encode a key (%XX) when it holds %, ?, #, a space or anything other than printable ASCII, or when it has an empty, . or .. part between slashes, like a//b or ../x: clients and proxies change those before the request arrives. Encoding every character that is not a letter, digit, -, _, . or ~ is always safe, so user/42 may also be written user%2F42. Any byte can be part of a key.
• The body is the value's bytes, exactly as written. It's the only response that isn't JSON.
• Isledb-Revision is the revision the reader answered from. Every write whose revision is at or below it is included.
• A missing key is 404 with {"error":"key_not_found",…}, still with Isledb-Revision: the key was not there at that revision. A key whose TTL has passed is missing.
• HEAD answers the same status and headers, without the body.

Scan keys

GET /v1/kv?prefix=user/&limit=2

200 OK
Content-Type: application/json
Isledb-Revision: 1042

{"items":[{"key":"user/1","value":"{\"name\":\"a\"}"},
          {"key":"user/2","value":"{\"name\":\"b\"}"}],
 "next_page":"AXVzZXIvMw"}

Choose the keys with prefix, or with start and end; leave all three out to scan every key:

For example, with the keys order/0001 to order/0500:

GET /v1/kv?start=order/0100&end=order/0200     order/0100 to order/0199
GET /v1/kv?start=order/0450                    order/0450 and every key after it
GET /v1/kv?end=order/0003                      every key before order/0003
GET /v1/kv?prefix=order/                       every key starting with order/

end itself is never included. To include it, add a zero byte: end=order/0200%00 stops right after order/0200.

• Items come in key order, sorted by their bytes.
• Each item is text or base64: key and value when they are valid UTF-8 text, else key_base64 and value_base64, the same fields a write batch takes. An empty value is "value":"".
• Get the next page by sending the same request with page set to next_page. No next_page means there are no more keys.
• A page may hold fewer than limit items even when more follow: it also stops at about 1 MiB of keys and values. It always holds at least one item.
• Each page reads the newest data and has its own Isledb-Revision. A key written or deleted while you page through may or may not be seen, but no key comes twice, and every key there for the whole scan is seen. A scan can move to another replica between pages.
• Values in query parameters are URL-encoded, so any byte can be part of a prefix or bound.

Follow the change feed

When the database has a change feed (the writer's change_feed.payload), readers serve it:

GET /v1/changes?from=oldest&limit=2

200 OK
Content-Type: application/json

{"changes":[{"sequence":41,"revision":1042,"op":"put","key":"user/42","value":"{\"name\":\"a\"}"},
            {"sequence":42,"revision":1043,"op":"delete","key":"user/7"}],
 "next":"cf1_AAAAAAAAAAIAAAAAAAAAAA",
 "caught_up":true}

• Changes come in the order they were written. Each has its sequence, its revision (the lowest revision that includes it, at or below the one the writer returned; changes written together share one), its op (put or delete), and its key as key or key_base64, as in a scan.
• A put carries its value as value or value_base64 when the feed keeps values; a keys-only feed sends only the key. A put with a TTL has expires_at.
• With a keys-only feed, check the revision when you read the key. A replica can still be behind the change. If the read's Isledb-Revision is below the change's revision, it does not hold the change yet: read again shortly. A 404 below that revision means the same, not that the key is gone.
• Keep your place with next. After applying a page, save its next and send it as from for the following page. A cursor works on any reader replica. Choose your start once, save the first next before applying anything, and from then on always resume from your saved cursor: starting again from head after a restart would skip what changed meanwhile.
• caught_up: true means there is nothing newer yet. Poll again with the same next, for example every second. A reader answers a caught-up poll from what it has loaded, without reading storage, so polling often costs nothing. New changes reach a poll within about a second of being written (about two on Google Cloud Storage).
• head is about a second behind, like any read: a consumer starting at head may also get the changes of the last second.
• A page may hold fewer than limit changes, even none, while more follow: it also stops at about 1 MiB of keys and values, and some writes record no changes. Keep reading until caught_up is true.

When your cursor is too old. If the database's change-feed retention has deleted changes after your cursor, the answer is 410 with {"error":"cursor_expired","oldest":"<cursor>",…}. Either continue from oldest, accepting the gap, or rebuild your copy of the data (below).

Start from the data, then follow the feed. To build a copy of the database and keep it current:

1. Read GET /v1/changes?from=head and save its next.
2. Copy every key with a scan.
3. Follow the feed from the saved cursor, applying each put as "set this key" and each delete as "remove this key".

Changes made during the scan are applied again, but the feed holds every later change too, so the copy ends up right.

How fresh a read is

A reader answers from the newest data it has loaded. It loads the latest data about once a second, or, when it reads through an isledb cache, usually within a second of each write. So a read can miss a write made in the last second or so. On Google Cloud Storage, which takes at most one write a second to the database, allow about two seconds.

If a reader can't reach storage, it keeps answering from the data it has until that data is max_pinned_view_age old (1 hour by default), and only then answers 503. So during a storage outage, reads can be up to that old.

Responses carry Cache-Control: no-cache, so caches in front of the readers check before reusing an answer.

To check whether a read includes a particular write, compare its Isledb-Revision with the revision the writer returned: a read at or past that revision includes it.

Errors

Error responses are JSON:

{"error":"key_not_found","message":"no such key"}