Embedded key-value database for Go

A key-value database for object storage.

Store durable key-value data directly in object storage. Readers scale independently; compaction runs on separate compute.

Get started See the architecture
Why IsleDB

Use your bucket as the durable home for key-value data.

IsleDB is an embedded Go library. Your compute can come and go while the database stays durable, shared, and available through the object store you already operate.

01

Object-store durability

Keep the complete database under one bucket prefix instead of operating a separate storage fleet.

02

Readers scale independently

Add reader processes for lookups, scans, snapshots, and change consumption without moving the dataset.

03

Compute stays flexible

Run writers, readers, and maintenance as separate services, scheduled jobs, or ephemeral workers.

Architecture

One writer. Many readers. Independent compaction.

One writer commits updates to shared object storage. Any number of readers open the same database. Maintenance runs independently on the compute you choose.

Write

One writer

Commits key-value updates.

Store

Your object store

Holds durable shared database state.

Read

Many readers

Scale without moving the data.

Separate maintenance worker Runs compaction, optional change-history retention, and storage cleanup. continuous or scheduled
Compaction

Run compaction on separate compute.

A typical embedded LSM engine compacts on the same machine that serves application traffic. IsleDB lets a dedicated maintenance process perform that work through shared object storage.

01Keep application CPU and memory focused on serving traffic.
02Scale maintenance resources independently from writers and readers.
03Run continuously or schedule compaction for quieter periods.
m, err := db.OpenMaintenance(ctx,
  isledb.DefaultMaintenanceOptions(),
)
if err != nil { return err }

runErr := m.Run(ctx)

closeCtx, cancel := context.WithTimeout(
  context.Background(), 30*time.Second,
)
defer cancel()
closeErr := m.Close(closeCtx)

return errors.Join(runErr, closeErr)
Typical embedded LSM

Application and compaction share a machine

Reads, writes, and background compaction compete for the same local resources.

IsleDB

Compaction gets its own process

The maintenance worker can run elsewhere while application processes continue serving traffic.

Go API

A small API with explicit boundaries.

Open a database for one bucket prefix. Flush writer updates when they should become visible, refresh readers when immediate freshness matters, and run maintenance where it fits your deployment.

DB Writer Reader Snapshot ChangeReader Maintenance
main.go 
// Open a database in one object-store prefix.
db, err := isledb.Open(ctx,
  "s3://my-bucket?region=us-east-1",
  isledb.DBOptions{Prefix: "prod/users"},
)
if err != nil { return err }
defer db.Close()

w, err := db.OpenWriter(ctx,
  isledb.DefaultWriterOptions(),
)
if err != nil { return err }
defer w.Close(ctx)

if err := w.Put(ctx, []byte("user:42"), profile); err != nil {
  return err
}
if err := w.Flush(ctx); err != nil { return err }

r, err := db.OpenReader(ctx,
  isledb.DefaultReaderOpenOptions("./cache"),
)
if err != nil { return err }
defer r.Close()

value, found, err := r.Get(ctx, []byte("user:42"))

Keys only

invalidation

Identify the changed key and operation without storing another copy of each PUT value.

Full values

replayable CDC

Include committed PUT values when consumers need to reconstruct historical mutations.

Optional change feed

Consume committed changes at your own pace.

Each consumer chooses a starting point, reads an ordered page with count and byte limits, saves its position, and resumes later.

Ordered puts and deletes
Bounded pages
Independent consumers
Object storage backends

One Go key-value database across S3, GCS, and Azure Blob Storage.

Deploy on AWS, Google Cloud, Azure, private S3-compatible storage, or local files without changing how your application writes, reads, scans, or consumes changes.

AWS + compatible

Key-value database on Amazon S3

Keep the database under one S3 bucket prefix or use an S3-compatible service.

Google Cloud

Key-value database on GCS

Use Google Cloud Storage for durable KV data shared by many readers.

Microsoft Azure

Key-value database on Azure Blob

Store the database in an Azure Blob container with the same Go API.

Private + local

MinIO and file-backed storage

Develop locally or use an S3-compatible private cloud with one data model.

Workload fit

Choose object storage on purpose.

IsleDB favors large, durable, append-shaped workloads where shared storage and independently scalable compute matter more than single-digit millisecond latency.

Strong fit

  • Large retained key-value state
  • Append-heavy events, logs, and CDC
  • Many readers over one durable dataset
  • Ephemeral or independently scaled compute
  • Second-scale freshness

Choose something else

  • Sub-10ms read-after-write serving
  • High-contention transactional updates
  • Joins, secondary indexes, or SQL queries
  • Multi-key ACID transactions
  • Tiny hot datasets that fit in memory
Key-value database FAQ

Common questions about KV databases on object storage.

IsleDB is an embedded Go library that keeps durable database state in your bucket.

What is a key-value database on object storage?

It retrieves values by key while using an object or blob store as the durable storage layer.

Is IsleDB a key-value database for Go or Golang applications?

Yes. It runs inside Go applications and provides native writer, reader, change-feed, and maintenance APIs.

Can IsleDB run compaction in another process?

Yes. A maintenance process can perform compaction on separate compute against the shared database.

Does IsleDB work with S3, GCS, and Azure Blob Storage?

Yes. Choose the matching bucket URL and credentials while keeping the same database API.

When should I use a different database?

Choose a transactional serving database for sub-10ms latency, joins, or multi-key transactions.

Get started

Build a Go key-value database on your object-store bucket.

Write your first key, add readers when you need them, and run compaction wherever it fits.

View on GitHub API reference