AuroraMeter. Events. Replay
(Aurora Meter v1.0.0)
View Source
Rebuilds aurora_meter_event_totals from the events that produced it, into
an isolated generation, while the system keeps recording.
{:ok, report} = AuroraMeter.Events.Replay.run()
AuroraMeter.Events.Replay.status()
AuroraMeter.Events.Replay.prune(1)A replay is an operator action, not a scheduled job. Nothing runs one by
itself: AuroraMeter.Oban schedules no replay, and the runbook
(replay) says when to run one and what it costs.
What makes a forward scan sound here
The events table is being written while the scan reads it, and two obvious
keyset orders are unsound: id is a random version 4 UUID, so a transaction
that started earlier can commit a row below a cursor the scan has passed, and
inserted_at is an application clock value rather than a commit order. The
scan orders by seq, the database-assigned identity column, and is bounded
by a watermark:
AuroraMeter.Storage.begin_projection_generation/0takesFOR UPDATEon theevents_projectioncheckpoint row. Every record and correction transaction takesFOR SHAREon that same row before inserting anything, so the exclusive lock is granted only once every in-flight one has finished.max(seq)read in that transaction is therefore a line every committed event is below.- Every record that starts afterwards reads the building generation in its own first step and writes its delta to both generations. Those events are above the watermark and the scan never sees them.
There is no third case, which is what makes "no event is missing and none is counted twice" provable rather than likely.
What a replay never does
It reads aurora_meter_events and writes aurora_meter_event_totals and its
own checkpoint rows. It never calls AuroraMeter.record/4, so it stages no
export intent and inserts no event. It never grants a credit, never marks a
counter key dirty, never writes pending_flush, never publishes a PubSub
message and never calls a host handler. A rebuilt projection therefore cannot
cause a resend, a second invoice line or a duplicate notification (I08).
Exclusion, and why there is no lease
Two runners are kept apart by AuroraMeter.Checkpoints.claim/3, a Postgres
session advisory lock on a pinned connection. It dies with the connection, so
a runner killed with kill -9 leaves nothing to expire. The heartbeat in the
checkpoint cursor is a report for a human and nothing subtracts it from
anything: open-findings.md X100 measured the one clock every node shares
stepping backwards 439 ms on a 32.5 second cadence, and a lease compared
against it can invert.
Repo-bound, like the backfill
Each batch commits its totals and its checkpoint in one transaction, which is
the whole of the resumability guarantee, so this module opens that
transaction on AuroraMeter.Config.repo/0 directly, exactly as
AuroraMeter.Checkpoints and AuroraMeter.Events.Backfill do. An adapter
that does not declare :projection_generations is refused up front with
{:error, {:unsupported, :projection_generations}} rather than half served.
Summary
Functions
The checkpoint row name a build of generation keeps.
The name of the advisory-lock claim a run holds, shared by every generation.
Deletes a retired generation's rows in bounded slices.
Rebuilds the projection into a new generation and, by default, activates it.
The generations and, when one is in progress, the replay's own checkpoint.
Types
@type report() :: %{ generation: pos_integer(), scanned: non_neg_integer(), projected: non_neg_integer(), keys: non_neg_integer(), batches: non_neg_integer(), differences: non_neg_integer(), activated: boolean(), duration_ms: non_neg_integer() }
What a completed run reports.
@type status() :: %{ active_generation: non_neg_integer(), building_generation: integer() | nil, previous_generation: non_neg_integer() | nil, seed_generation: integer() | nil, watermark: non_neg_integer() | nil, replay: map() | nil }
What status/0 reports.
Functions
The checkpoint row name a build of generation keeps.
Examples
iex> AuroraMeter.Events.Replay.checkpoint_name(2)
"events_replay:2"
@spec claim_name() :: String.t()
The name of the advisory-lock claim a run holds, shared by every generation.
Two replays of different generations exclude each other too: a second building generation while one is in flight would leave the record path writing three rows and no way to say which build a delta belonged to.
Examples
iex> AuroraMeter.Events.Replay.claim_name()
"events_replay"
@spec prune(integer()) :: {:ok, non_neg_integer()} | {:error, :active | :not_found | {:already_running, status()}} | {:error, term()}
Deletes a retired generation's rows in bounded slices.
Refuses the active generation ({:error, :active}), refuses while a replay
holds the claim ({:error, {:already_running, status}}) and refuses a
generation with no rows and no mention in the projection state
({:error, :not_found}).
Pruning the generation a replay abandoned also drops its seed rows and clears
it from the projection state, so the next run/1 starts a fresh build.
Examples
iex> AuroraMeter.Events.Replay.prune(0)
{:error, :active}
@spec run(keyword()) :: {:ok, report()} | {:ok, :paused, status()} | {:error, {:projection_mismatch, map()}} | {:error, {:already_running, status()}} | {:error, term()}
Rebuilds the projection into a new generation and, by default, activates it.
Options:
:batch_size: events per committed batch, default 5000.:compare::require_match(default) refuses to activate when the rebuilt generation differs from the live one and returns{:error, {:projection_mismatch, summary}}, leaving the built generation for inspection.:reportreturns the differences and activates anyway, which is the mode for repairing a projection already known to be wrong.:activate: defaulttrue.falsebuilds and compares only.:resume: defaulttrue.falserefuses when a building generation already exists rather than continuing it.:generation: refuse unless the building generation is this one. For an operator who wants to be sure which build they are resuming.:compare_limit: differences listed, default 1000. The count is always complete.:max_batches: stop cleanly after this many committed batches, leaving the cursor where it is. For a bounded maintenance window.:rehydrate: defaulttrue. Re-seats warm events-source counter keys after activation so dashboards move without a restart.:timeout: milliseconds for one statement and one batch transaction, default 60000.
A paused replay returns {:ok, :paused, status} at its next batch boundary
with everything in place; AuroraMeter.Checkpoints.resume/1 and another
run/1 carry on from the cursor.
@spec status() :: status()
The generations and, when one is in progress, the replay's own checkpoint.
Examples
iex> status = AuroraMeter.Events.Replay.status()
iex> is_integer(status.active_generation)
true