SuperVersion wrapper around the LSM tree's sstables list with enter/exit_ ref counting would be a high-value additionDate: 2026-05-29
Time: 09:59
I have enough from the observations to give a thorough explanation. Let me work with what's already gathered.
log-structured-merge-tree/lsm.py:compact — The compaction method where the race condition manifests; lines 321–352 show the unprotected swap-and-delete sequencewrite-ahead-log/wal.py — The only module in this repo using self._lock for thread safety; compare its discipline with the LSM tree's lack thereofsstable-and-compaction/sstable.py — The CompactionManager class (line 301+) takes a different approach to SSTable lifecycle management; get_sstables() returns a copy but doesn't solve the file-deletion racerocksdb-superversion-design — RocksDB's SuperVersion struct in db/column_family.h is the canonical implementation of this pattern; studying it reveals how ref counting interacts with flush, compaction, and iterator pinningmvcc-for-metadata-vs-data — DDIA discusses MVCC for user-visible data, but the same principle applies to engine-internal structures; this is the conceptual bridge between Chapter 7 (transactions) and Chapter 3 (storage engines)lsm-sstables-unprotected-mutation — LSMTree.sstables is mutated by both flush() (append) and compact() (replace) with no synchronization or versioning, making concurrent reads unsafecompaction-deletes-before-reader-release — compact() deletes old SSTable files immediately after replacing self._sstables, with no mechanism to defer deletion until active readers finishsstable-compaction-manager-copies-list — CompactionManager.getsstables() returns a shallow copy of sstables, preventing iterator invalidation but not file-deletion raceswal-module-uses-locking-lsm-does-not — The WAL module in write-ahead-log/wal.py uses self._lock for thread safety while the LSM tree module has no locking or concurrency control