WalShadow: Physical WAL Replication to ClickHouse

WalShadow aims for sub-second replication from Postgres to ClickHouse by reading the physical Write-Ahead Log (WAL) instead of relying on logical replication slots.

The Technical Shift

Logical replication is the standard, but it's slow. It requires the source DB to decode the WAL into a logical format, which consumes CPU and can cause "replication slot bloat" if the consumer lags, potentially filling up the source disk. WalShadow reads the physical blocks—effectively "shadowing" the disk state—and transforms them into ClickHouse inserts. It's essentially a Change Data Capture (CDC) tool that treats the WAL as a raw byte stream.

The "Catch": The Transformation Overhead

The cost is moved from the source CPU to the transformation layer. Because physical WAL contains page-level changes, not row-level changes, WalShadow has to reconstruct the rows. This requires a deep understanding of the Postgres page layout. If you change your Postgres version or use non-standard page sizes, your replication pipeline breaks. You've traded "logical stability" for "physical speed."

Who Should Skip

Teams with highly volatile schemas or those using managed Postgres services (like RDS) where you don't have raw access to the physical WAL files. If you can't "touch the disk," WalShadow is a non-starter.

Verdict

A high-performance niche tool. It's the "Formula 1" of replication: incredibly fast, but requires a dedicated pit crew to keep it from crashing when the environment changes.