File: multi-leader-replication/testertestmulti_leader.py

Date: 2026-05-29

Time: 13:54

Purpose

This is the standalone test harness for the multi-leader replication module. It validates that the multi_leader.py implementation correctly handles the core challenges of multi-leader replication as described in DDIA: conflict resolution, topology-aware propagation, convergence, and idempotent sync.

The tester prefix distinguishes it from testmultileader.py (likely a pytest-compatible suite). This file is self-contained — it runs directly via python testertestmultileader.py with a manual test runner at the bottom, producing pass/fail output and setting the exit code.

Key Components

Test Functions

| Test | What it validates |

|------|-------------------|

| testbasicreplicationnoconflict | Non-conflicting writes across 3 nodes converge after one sync, with zero conflict records |

| testlwwconflicthighertimestamp_wins | LWW resolves concurrent writes by choosing the higher Lamport timestamp |

| testlwwtiebreakbynodeid | When timestamps tie, the lexicographically higher nodeid wins — deterministic tiebreak |

| testcustommerge | A user-supplied merge function (counter addition) replaces LWW for domain-specific resolution |

| testtombstonedelete | Deletes propagate as tombstones; get() returns None after sync |

| testringtopology | Ring topology requires >=2 sync rounds (hop-by-hop) vs. all-to-all |

| testconflictlogging | ConflictRecord captures key, both values, and the strategy used |

| testlamportclock_ordering | Local writes produce monotonically increasing timestamps; remote apply advances the clock past the remote timestamp |

| test_idempotency | Double-syncing is a no-op — no duplicate entries or spurious conflicts |

| testconvergencemany_keys | 5 nodes × 20 keys each all converge after sync |

| testcustommergerepeatedsync_converges | Custom merge doesn't "explode" on repeated syncs (e.g., counter doesn't keep doubling) |

Manual Test Runner (lines 147–163)

A _main block that iterates over all test functions, catches exceptions to report PASS/FAIL, and exits with code 1 on any failure. This pattern appears across all tester*.py files in the repo — it's the project's convention for CI-friendly test execution without pytest.

Patterns

Cluster-as-test-fixture: Every test instantiates a MultiLeaderCluster with a list of node IDs. The cluster manages node lifecycle, topology wiring, and sync orchestration. Tests never manually wire nodes together.

Sync-then-assert: The dominant pattern is write → cluster.sync() → assert convergence. sync() is a discrete step, not background — tests control exactly when replication happens.

Convergence oracle: cluster.all_converged() provides a single boolean check that all nodes agree on all keys. Tests use this as the primary correctness assertion alongside spot-checks on specific keys.

Strategy injection: Conflict resolution strategy is passed at cluster construction time (ConflictStrategy.CUSTOMMERGE + mergefn), not per-operation. This mirrors DDIA's model where conflict strategy is a system-level policy.

Dependencies

Imports from multi_leader:

Imported by: Nothing — this is a leaf test file.

Flow

1. Each test function constructs a cluster with specific parameters (node count, topology, strategy).

2. Writes happen via cluster.node(id).put(key, value), which records the change locally with a Lamport timestamp.

3. cluster.sync() triggers one round of replication across the configured topology. For ring topology, cluster.syncuntilconverged() loops sync rounds until convergence.

4. Assertions check both individual key values and global convergence.

5. The _main_ runner collects results and produces a summary line like 11 passed, 0 failed.

Invariants

Error Handling

Minimal — this is a test file. Each test function either returns (pass) or raises AssertionError (fail). The _main_ runner catches all exceptions via a bare except Exception, prints the failure, and continues to the next test. Exit code 1 signals any failure to the caller.

There's no retry logic, no cleanup, and no teardown — clusters are ephemeral in-memory objects that get garbage collected.

Topics to Explore

Beliefs