Abhishek Kumar← back
Case study

Setu

A hybrid data-migration platform that moves data where the data lives.

GoNext.jsPostgreSQLRiverS3
The problem

Enterprises must move data between many sources and destinations — but a lot of that data lives on-prem or at the edge, behind firewalls, where you can't simply pull it into the cloud.

The approach

A control-plane / data-plane split. One streaming ETL engine is written once and runs identically in the cloud AND on a tiny, outbound-only edge agent — so a migration can execute right next to the data.

Key decisions

Outbound-only edge agent

No inbound ports; a long-poll job protocol. Edge secrets are resolved on-host and never sent to the cloud. Per-connection cloud/agent placement.

Atomic job leasing

Postgres FOR UPDATE SKIP LOCKED, with a heartbeat + lease-TTL reaper and exponential backoff so jobs never double-run or get stuck.

Durable runs

A River queue makes runs resumable and cancellable, with SSE live logs.

Pluggable blob seam

Filesystem or S3, with automatic direct-vs-staged execution, cursor-based incremental watermarks, and cron scheduling.

Results