Date: 2026-05-29
Time: 10:22
testdynamotester.py is a QA-level integration test suite for the Dynamo-style leaderless replication module. It sits alongside the developer-written testdynamo.py and serves as an independent validation layer — the "tester" naming convention across this repo (testertest*.py or test*_tester.py) indicates tests generated or curated to verify the implementation against its spec, distinct from the author's own unit tests.
This file exercises the full public API of dynamo.py: cluster formation, quorum reads/writes, node failure handling, read repair, conflict detection, sloppy quorums with hinted handoff, and anti-entropy repair.
The file defines 10 test functions, each targeting a specific behavioral contract:
| Test | What it validates |
|------|-------------------|
| testspecexample_full | End-to-end happy path: write, read, node failure, quorum enforcement, read repair, anti-entropy |
| testspecsloppyquorumexample | Sloppy quorum stores hints on live nodes and delivers them when the target recovers |
| testreadnonexistent_key | Reading a missing key returns None/version 0 without conflict |
| testversionrollbackonfailed_write | A failed quorum write must not increment the version counter — the next successful write picks up from the last committed version |
| testnodeunavailablerejectsoperations | ReplicaNode rejects reads/writes when marked unavailable, preserving prior state |
| testmultiplekeys_independent | Version counters are per-key, not global |
| testconflictdetection | When replicas hold the same version number but different values, a read returns is_conflict=True with all conflicting values as a list |
| testlargecluster10kops | N=7 cluster handles 10k writes across 50 keys without error (smoke/perf test) |
| testantientropyfullsync | Anti-entropy repair synchronizes a node that missed all writes while offline |
| testnohintswithoutsloppyquorum | With sloppyquorum=False, no hints are stored or delivered — offline nodes stay empty until anti-entropy runs |
Failure injection via setnodeavailable: Every test that exercises fault tolerance follows the same pattern — take nodes down, perform operations, bring nodes back, assert repair behavior. This is a controlled simulation of network partitions.
Direct store manipulation for conflict injection (test 7): Rather than orchestrating a real concurrent-write scenario, the test injects divergent VersionedValue entries directly into _store on each node. This is a pragmatic shortcut — it tests conflict *detection* without requiring the implementation to produce conflicts organically.
Quorum arithmetic as a test parameter: Tests carefully choose writequorum and readquorum values to create the exact failure conditions needed. For instance, testversionrollbackonfailed_write uses W=3, R=1 so that a single node failure guarantees write failure.
Naming convention: The file is testdynamotester.py (not testertestdynamo.py like most other modules), which is a minor inconsistency in the repo's naming scheme.
Imports from dynamo.py:
DynamoCluster — the cluster facade (primary entry point)QuorumNotMet — exception raised when insufficient replicas respondReplicaNode — individual storage node (tested directly in test 5)VersionedValue — data class for value + version + origin node (used to inject conflicts in test 7)ReadResult — return type of cluster.get() with .value, .version, .isconflict, .replicasrepairedNothing imports this file — it's a leaf test module.
Tests follow a consistent structure:
1. Construct a DynamoCluster with specific quorum parameters
2. Setup — write initial data, optionally take nodes offline
3. Act — perform the operation under test (put, get, repair, deliver hints)
4. Assert — verify return values, side effects on specific nodes, or exception behavior
Test 1 (testspecexample_full) is the most complex, walking through an entire lifecycle: write → read → node failure → write with degraded quorum → node recovery → read repair → quorum failure → anti-entropy.
These tests collectively enforce:
W successful replica writes; fewer raises QuorumNotMet. Reads require R responses."a" twice gives version 2; key "b" written once stays at version 1.replicas_repaired >= 1 without requiring explicit action.is_conflict=True and return all conflicting values as a list.sloppy_quorum=True. Without it, offline nodes receive nothing until anti-entropy runs.QuorumNotMet is the only exception tested. It is expected (via pytest.raises) when too many nodes are down to satisfy the write quorum.False from write() and None from read() — failures are values, not exceptions, at the node level. The cluster layer aggregates these into quorum decisions.leaderless-replication/dynamo.py — The implementation under test; understanding DynamoCluster.put/get internals explains why the quorum arithmetic in these tests worksleaderless-replication/dynamo.py:antientropyrepair — How the repair mechanism discovers and resolves inconsistencies across replicasleaderless-replication/test_dynamo.py — The developer-written tests; comparing coverage with this tester file reveals what each suite uniquely validatesquorum-intersection-guarantee — Why R + W > N ensures at least one node in a read set has the latest write — the theoretical foundation for every quorum parameter choice in these testssloppy-quorum-tradeoffs — How sloppy quorums trade consistency for availability, and why hinted handoff is necessary to eventually convergedynamo-version-rollback-on-failed-write — A failed quorum write must not increment the version counter; the next successful write uses the version that would have been assigned to the failed onedynamo-conflict-detection-same-version-different-values — When multiple replicas hold different values at the same version number, ReadResult.is_conflict is True and value is a list of all conflicting valuesdynamo-sloppy-quorum-opt-in — Hinted handoff only occurs when DynamoCluster is constructed with sloppyquorum=True; without it, deliverhints() returns 0 and offline nodes stay emptydynamo-per-key-versioning — Version counters are scoped per key, not global across the cluster; independent keys maintain independent version sequencesdynamo-read-repair-on-get — A cluster.get() call that discovers stale replicas automatically repairs them and reports the count via ReadResult.replicas_repaired