Skip to content
Three stacked strata forming a ridge, the Moraine markThree stacked strata forming a ridge, the Moraine mark

Moraine

Run DuckLake entirely in object storage, no server required.

DuckLake stores table data as Parquet files in object storage and hosts its catalog in a SQL database: SQLite, Postgres, or DuckDB (served over Quack). The database layer provides transactional consistency and ACID guarantees, but requires hosting an additional server.

Moraine removes the need for a separate catalog database. Instead, it serves the catalog from SlateDB, an embedded transactional key-value store that also lives in object storage. That means that the catalog resides in the same bucket as the data files. In addition, Moraine is the only DuckLake extension that offers DuckDB native indexes for faster querying.

LOAD 'moraine.v2.0.0.linux_amd64.duckdb_extension';

CREATE SECRET aws_secret (TYPE s3, KEY_ID '…', SECRET '…', REGION 'us-west-2');
ATTACH 'ducklake:moraine:s3://bucket/lake' AS lake
  (DATA_PATH 's3://bucket/lake-data/', READ_WRITE);

CREATE TABLE lake.events (id BIGINT, payload VARCHAR);
INSERT INTO lake.events VALUES (1, 'hello, bucket'), (2, 'hello again');
SELECT * FROM lake.events;

CREATE UNIQUE INDEX events_by_id ON lake.events (id);
SELECT payload FROM lake.events WHERE id = 2;
  • No catalog server: No catalog server to deploy or monitor. The only permanent resource is a storage bucket.
  • Nothing to pay for while idle: An idle lake costs only storage, no compute or database hosting fees.
  • Bucket durability: The catalog is objects in the bucket, so it inherits the bucket’s durability, replication, and versioning. A copy taken while no writer is attached is a complete lake; see Backups and copies.
  • Indexed lookups: Ordered indexes over table columns, created with CREATE INDEX and maintained in the same commit as the rows, so a query by key probes the index instead of scanning every file; see Index a column.
  • Sub-5 ms commits. A commit is durable once one write-ahead-log PUT lands: 50 to 100 ms on S3 Standard, 5 to 10 ms with the log on S3 Express One Zone via META_WAL_PATH.
  • Many independent writer processes. One process writes; the rest attach READ_ONLY. See one writer, many readers.
  • INSTALL moraine FROM community. The listing is in progress. Until then, signed builds come from Moraine’s own extension repository, which takes one CREATE EXTENSION REPOSITORY to register.

Identical DuckLake workloads through the same DuckDB CLI, only the catalog swapped, each on the store it would really use: Moraine on S3, the same with its write-ahead log on S3 Express, Postgres on RDS, and a DuckDB catalog file on local disk, since that is its only option.

Moraine Moraine, log on Express DuckDB file Postgres
20 single-row commits, one connection 823 ms 305 ms 209 ms 269 ms
commits per second, one process, 1 connection 30 73 125 87
commits per second, one process, 8 connections 186 396 83 37
commits per second, one process, 16 connections 353 563 74 26
attach 604 ms 615 ms 16 ms 189 ms
read a count once attached 28 ms 28 ms 31 ms 72 ms
full scan 135 ms 131 ms 37 ms 75 ms

A commit is one object write, the write-ahead-log entry that makes it durable, so its cost is that round trip: about 28 ms on S3 and 15 ms with the log on S3 Express. That is why one connection is slower here than a catalog on a local file or a nearby Postgres, and why there is no server to run. Attach is a start-up cost, paid once by a process that stays attached. Moraine is single-writer across processes — a second writer process is fenced — and readers are not limited that way. Connections within one process share a flush, so when they overlap, throughput rises where the others’ falls.

For more details, see Benchmarks.

Moraine is under active development and in production use at contour.app. All DuckLake SQL, including CREATE, INSERT, UPDATE, DELETE, time travel, and maintenance, runs against Moraine.

Dual-licensed under MIT or Apache-2.0.

More details

  • Getting started: how to install the extension and attach a lake.
  • Configuration: attach options, environment variables, and DuckDB functions.
  • Operating manual: retention, maintenance, encryption, backups, and upgrades.
  • Comparison: Moraine alongside DuckDB, Postgres, Quack, Iceberg, and Delta Lake.
  • Design: key design decisions and rationale.
  • Benchmarks: performance benchmarks against other DuckLake catalogs.
  • Releases: changelog, extension downloads, and the DuckDB compatibility table.
  • Roadmap: what’s next for Moraine.