Topic: DDIA Chapter 12's argument for decomposing databases into logs + derived views; the conceptual foundation for this implementation

Date: 2026-05-29

Time: 11:35

Now I have a thorough understanding of the codebase. Let me write the explanation.

Topics to Explore

• [function] unbundled-database/unbundled_database.py:StorageEngine.rebuild — Proves the storage engine is truly derived: destroy it and reconstruct entirely from the WAL to see that the log, not the store, is the source of truth
• [function] unbundled-database/unbundleddatabase.py:CDCStream.snapshotand_stream — The catch-up mechanism for late-joining consumers; compare with Kafka's compacted-topic bootstrapping approach
• [function] event-sourcing-store/eventstore.py:Projection.catchup — Compare with DerivedSystem.process_event to see how event sourcing and CDC converge on the identical position-tracking pattern
• [general] flush-semantics-across-modules — The unbundled database requires explicit flush() (lazy propagation) while event sourcing uses synchronous subscriber notification (LiveProjection) — explore the consistency/latency tradeoffs of push vs. pull delivery
• [file] stream-join-processor/streamjoinprocessor.py — The watermark and deferred-miss-emission logic implements the hard part of stream processing: deciding when an unmatched event will *never* match

---

Beliefs

• wal-is-source-of-truth — StorageEngine has no direct mutation API; all state changes flow through WriteAheadLog.append → StorageEngine.apply, and rebuild(wal) reproduces identical state from the log alone
• cdc-old-value-required-for-index-consistency — SecondaryIndex.processevent depends on CDCEvent.oldvalue to remove stale index entries during updates; without before-images, incremental index maintenance produces phantom references
• derived-systems-independently-position-tracked — Each DerivedSystem tracks its own LSN position independently, allowing consumers to fall behind or catch up at different rates without coordinator state or blocking
• lazy-propagation-via-explicit-flush — Derived systems receive no CDC events until UnbundledDatabase.flush() is called; the gap between put() and flush() is observable as lag, providing predictable eventual consistency rather than strong consistency
• event-sourcing-and-cdc-converge-on-projection-pattern — Both Projection.catchup and DerivedSystem.processevent track a position cursor, pull/receive ordered events, and apply type-dispatched handlers — the same structural pattern solving "derived data" from different starting points