IsleDB is a Go database library for S3, GCS, Azure Blob, and MinIO, built for writes, scans, tailing readers, compaction, and object-storage-first workloads.
// Connect to object storage
store, _ := blobstore.Open(ctx, "s3://my-bucket?region=us-east-1", "db1")
db, _ := isledb.OpenDB(ctx, store, isledb.DBOptions{})
// Write
writer, _ := db.OpenWriter(ctx, isledb.DefaultWriterOptions())
writer.Put([]byte("hello"), []byte("world"))
writer.Flush(ctx)
// Read
reader, _ := isledb.OpenReader(ctx, store, isledb.ReaderOpenOptions{CacheDir: "./cache"})
val, ok, _ := reader.Get(ctx, []byte("hello"))
// Tail
tr, _ := isledb.OpenTailingReader(ctx, store, tailOpts)
tr.Tail(ctx, opts, func(kv isledb.KV) error { return nil })Memtable writes, immutable SST flushes, and reader-side caching provide predictable performance while your bucket handles durability and scale.
Writes land in the memtable. Large values can be stored as blobs.
Memtable is flushed into immutable SST files on object storage.
Capacity and durability scale with your bucket.
Stream new writes continuously, like tail -f for data.
IsleDB combines write/read APIs, compaction strategies, streaming tailing readers, TTL, and fence-based ownership.
Use S3, GCS, Azure Blob, MinIO, or local object-compatible storage.
Continuously consume new writes via polling and ordered replay.
Scale reads by adding more readers against the same prefix.
Merge compaction, age-based retention, and time-window retention.
Use TTL APIs to manage data lifetime without external schedulers.
Prevent split-brain writer/compactor ownership with manifest fencing.
One writer flushes SSTs, many readers serve queries, and tailers stream changes. Manifest fencing protects ownership during contention.
Memtable → SST → Upload
SSTs + blobs + manifest
Get · Scan · Tail
Buffers writes to reduce object-store PUT overhead.
Immutable sorted files that support efficient reads.
Tracks applied state and replay ordering.
Readers cache SST and blocks to avoid repeated downloads.
Core usage patterns in a compact view.
ctx := context.Background()
store, err := blobstore.Open(ctx, "s3://my-bucket?region=us-east-1", "db1")
db, err := isledb.OpenDB(ctx, store, isledb.DBOptions{})
defer db.Close()writer, _ := db.OpenWriter(ctx, isledb.DefaultWriterOptions())
defer writer.Close()
writer.Put([]byte("k"), []byte("v"))
writer.PutWithTTL([]byte("session"), data, 24*time.Hour)
writer.Flush(ctx)reader, _ := isledb.OpenReader(ctx, store, isledb.ReaderOpenOptions{CacheDir: "./cache"})
defer reader.Close()
value, ok, _ := reader.Get(ctx, []byte("k"))
rows, _ := reader.Scan(ctx, []byte("a"), []byte("z"))tr, _ := isledb.OpenTailingReader(ctx, store, isledb.TailingReaderOpenOptions{})
defer tr.Close()
_ = tr.Start()
_ = tr.Tail(ctx, isledb.TailOptions{}, func(kv isledb.KV) error {
return nil
})// Merge compactor for KV workloads
mergeCompactor, _ := db.OpenCompactor(ctx, isledb.DefaultCompactorOptions())
mergeCompactor.Start()
// FIFO compactor for event streams (keep recent data)
fifoCompactor, _ := db.OpenRetentionCompactor(ctx, isledb.RetentionCompactorOptions{
Mode: isledb.CompactByAge,
RetentionPeriod: 7 * 24 * time.Hour,
RetentionCount: 10,
})
fifoCompactor.Start()One storage-and-replay model powers event ingestion, state materialization, key-value APIs, and object-storage-first data pipelines.
Ingest app events and fan out with tailing readers.
Append ordered events and build projections from replay.
Serve Get/Scan workloads with object-store durability.
Stage changes in object storage before indexing and analytics.
Capture bronze data on object storage, then feed Iceberg/Delta/Hudi compaction and analytics pipelines.
Pick the right tool for the workload. IsleDB is strongest in object-storage-first, append-heavy systems.
"An isle is self-contained, durable, and reachable from anywhere. That's the model for your data on IsleDB."IsleDB philosophy — Your data island on object storage.
One go get, one bucket prefix, and a pragmatic API surface for write, read, replay, and stream.