Notes on replication lag

12 October 2026 · 6 min read

Replication lag is easy to measure badly. The number on the dashboard is usually either too noisy to alert on or too smoothed to notice a problem until the replica is minutes behind.

On a primary/replica setup, the tempting query is the one that subtracts the last replayed transaction timestamp from now:

SELECT now() - pg_last_xact_replay_timestamp() AS lag;

It is fine for a quick look, but it has two problems. On an idle primary there are no new transactions, so the value grows without bound even though the replica is perfectly caught up. And when the primary is busy, the timestamp only tells you about the last transaction that happened to be replayed, not about the work that is still queued.

Measure the right thing

What I actually want to know is: how far behind is the replica, in bytes and in time, and is that distance growing? On Postgres, the WAL positions give a more stable signal:

SELECT
  pg_wal_lsn_diff(pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn()) AS replay_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), pg_last_wal_replay_lsn())       AS total_lag_bytes;

Bytes are nice because they do not care whether the primary is idle. Time is still useful for humans, but it is better to derive it from a heartbeat table: write a row every second on the primary, replicate it, and measure the age of the newest row on the replica. That gives you a real end-to-end number, including the time spent waiting in the apply queue.

Alert on trends, not snapshots

A single high sample is usually a batch job or a network hiccup. What matters is whether the replica is catching up. I like a simple rule: page if the lag is above a threshold for five minutes, or if the derivative is positive for ten. The second condition catches the case where the absolute number is still small but the replica is falling behind steadily.

lag_p99 = histogram_quantile(
  0.99,
  rate(replication_lag_seconds_bucket[5m])
)

Keep the raw series too. A p99 is good for alerts, but when something breaks you want the ugly per-replica graph, not a smooth line that hides the outlier.

Backpressure is a product decision

Eventually the lag is large enough that reads from the replica are unsafe. The technical fix is easy to describe and hard to choose: route reads to the primary, reject writes, or serve stale data with a warning. Each one is a product decision. The important part is to decide it before the incident, and to make the behavior explicit in the code path that reads from the replica.

I have seen more incidents caused by an unbounded retry loop against a lagging replica than by the lag itself. A timeout and a fallback are usually worth more than another dashboard panel.

← Back to writing