Function: iterate in write-ahead-log/wal.py

Date: 2026-05-28

Time: 18:50

WriteAheadLog.iterate

Purpose

iterate provides raw, unfiltered access to every record stored in the WAL — including uncommitted batch operations, COMMIT markers, and CHECKPOINT records. This contrasts with replay, which filters out non-data records and skips uncommitted batches. It exists for diagnostic, inspection, or tooling scenarios where the caller needs to see the full on-disk state of the log, not just the logically committed view.

Contract

• Precondition: The WAL instance must be initialized (constructor completed successfully, so walfiles() and _fd are in a valid state).
• Postcondition: Yields every structurally valid record across all WAL segment files, in file order, stopping at the first corruption (CRC mismatch). No records are skipped or filtered.
• Invariant: The method does not modify any WAL state — it is a read-only scan. The flush before iteration ensures that any buffered writes from the current process are visible on disk.

Parameters

None beyond self. The method takes no arguments — it always scans the entire log from the beginning.

Return Value

Returns an Iterator[WALRecord]. Each WALRecord contains:

• seq_num: monotonically increasing sequence number
• op_type: one of "PUT", "DELETE", "COMMIT", "CHECKPOINT", or "UNKNOWN"
• key / value: UTF-8 decoded strings
• checksum: the CRC32 stored in the record

The caller must handle:

• An empty iterator (no WAL files or all files are empty)
• Records of any op type, including COMMIT and CHECKPOINT, which replay would suppress
• The iterator stopping abruptly at corruption — there is no signal distinguishing "end of log" from "stopped at bad record"

Algorithm

1. Acquire lock and flush: Grabs lock, checks if there's an open file descriptor (fd), and flushes it. This ensures any writes buffered in userspace by the current process are pushed to the OS before reading. The lock is released immediately after the flush — it is not held during iteration.

2. Delegate to readallrecords: Uses yield from to lazily stream records. readallrecords iterates over every .wal file in sorted filename order, opens each read-only, and calls readrecord in a loop until EOF (None return) or CRC failure (ValueError).

Side Effects

• I/O: Flushes the write fd (a flush() syscall — no fsync, so data reaches the OS page cache but isn't necessarily durable). Then opens each WAL segment file for reading during iteration.
• Lock scope: The lock is held only during the flush, not during the read phase. This means concurrent append calls can interleave with iteration. The iterator may or may not see records written after iterate() was called, depending on timing and whether those writes land in a file the iterator hasn't opened yet.

Error Handling

• CRC mismatch: readrecord raises ValueError on CRC failure. readall_records catches this and returns (stops iteration silently). The caller receives all records up to the corruption point with no indication that more data existed.
• File I/O errors (missing files, permission errors): Not caught — these propagate as OSError/IOError to the caller.
• Partial reads: If a record is truncated (incomplete write from a crash), readrecord returns None and iteration moves to the next file.

Usage Patterns

Typical callers use iterate when they need the raw log contents rather than the committed-only view:

wal = WriteAheadLog("/tmp/wal")
for record in wal.iterate():
    print(f"seq={record.seq_num} op={record.op_type} key={record.key}")

Caller obligations:

• Do not assume all yielded records are committed — partial batches (operations before a missing COMMIT) will appear.
• Do not assume the iterator covers the full log — corruption silently truncates it.
• Be aware that the iterator is not snapshot-isolated from concurrent writes.

Dependencies

• readall_records (internal): does the actual file scanning and record parsing
• walfiles (internal): returns sorted segment file paths
• readrecord (module-level): binary deserialization and CRC verification
• threading.Lock: for the flush synchronization
• Standard library: os, struct, zlib (indirectly, through readrecord)