AuroraMeter.Events.Replay (Aurora Meter v1.0.0)

View Source

Rebuilds aurora_meter_event_totals from the events that produced it, into an isolated generation, while the system keeps recording.

{:ok, report} = AuroraMeter.Events.Replay.run()
AuroraMeter.Events.Replay.status()
AuroraMeter.Events.Replay.prune(1)

A replay is an operator action, not a scheduled job. Nothing runs one by itself: AuroraMeter.Oban schedules no replay, and the runbook (replay) says when to run one and what it costs.

What makes a forward scan sound here

The events table is being written while the scan reads it, and two obvious keyset orders are unsound: id is a random version 4 UUID, so a transaction that started earlier can commit a row below a cursor the scan has passed, and inserted_at is an application clock value rather than a commit order. The scan orders by seq, the database-assigned identity column, and is bounded by a watermark:

  • AuroraMeter.Storage.begin_projection_generation/0 takes FOR UPDATE on the events_projection checkpoint row. Every record and correction transaction takes FOR SHARE on that same row before inserting anything, so the exclusive lock is granted only once every in-flight one has finished. max(seq) read in that transaction is therefore a line every committed event is below.
  • Every record that starts afterwards reads the building generation in its own first step and writes its delta to both generations. Those events are above the watermark and the scan never sees them.

There is no third case, which is what makes "no event is missing and none is counted twice" provable rather than likely.

What a replay never does

It reads aurora_meter_events and writes aurora_meter_event_totals and its own checkpoint rows. It never calls AuroraMeter.record/4, so it stages no export intent and inserts no event. It never grants a credit, never marks a counter key dirty, never writes pending_flush, never publishes a PubSub message and never calls a host handler. A rebuilt projection therefore cannot cause a resend, a second invoice line or a duplicate notification (I08).

Exclusion, and why there is no lease

Two runners are kept apart by AuroraMeter.Checkpoints.claim/3, a Postgres session advisory lock on a pinned connection. It dies with the connection, so a runner killed with kill -9 leaves nothing to expire. The heartbeat in the checkpoint cursor is a report for a human and nothing subtracts it from anything: open-findings.md X100 measured the one clock every node shares stepping backwards 439 ms on a 32.5 second cadence, and a lease compared against it can invert.

Repo-bound, like the backfill

Each batch commits its totals and its checkpoint in one transaction, which is the whole of the resumability guarantee, so this module opens that transaction on AuroraMeter.Config.repo/0 directly, exactly as AuroraMeter.Checkpoints and AuroraMeter.Events.Backfill do. An adapter that does not declare :projection_generations is refused up front with {:error, {:unsupported, :projection_generations}} rather than half served.

Summary

Types

What a completed run reports.

What status/0 reports.

Functions

The checkpoint row name a build of generation keeps.

The name of the advisory-lock claim a run holds, shared by every generation.

Deletes a retired generation's rows in bounded slices.

Rebuilds the projection into a new generation and, by default, activates it.

The generations and, when one is in progress, the replay's own checkpoint.

Types

report()

@type report() :: %{
  generation: pos_integer(),
  scanned: non_neg_integer(),
  projected: non_neg_integer(),
  keys: non_neg_integer(),
  batches: non_neg_integer(),
  differences: non_neg_integer(),
  activated: boolean(),
  duration_ms: non_neg_integer()
}

What a completed run reports.

status()

@type status() :: %{
  active_generation: non_neg_integer(),
  building_generation: integer() | nil,
  previous_generation: non_neg_integer() | nil,
  seed_generation: integer() | nil,
  watermark: non_neg_integer() | nil,
  replay: map() | nil
}

What status/0 reports.

Functions

checkpoint_name(generation)

@spec checkpoint_name(integer()) :: String.t()

The checkpoint row name a build of generation keeps.

Examples

iex> AuroraMeter.Events.Replay.checkpoint_name(2)
"events_replay:2"

claim_name()

@spec claim_name() :: String.t()

The name of the advisory-lock claim a run holds, shared by every generation.

Two replays of different generations exclude each other too: a second building generation while one is in flight would leave the record path writing three rows and no way to say which build a delta belonged to.

Examples

iex> AuroraMeter.Events.Replay.claim_name()
"events_replay"

prune(generation)

@spec prune(integer()) ::
  {:ok, non_neg_integer()}
  | {:error, :active | :not_found | {:already_running, status()}}
  | {:error, term()}

Deletes a retired generation's rows in bounded slices.

Refuses the active generation ({:error, :active}), refuses while a replay holds the claim ({:error, {:already_running, status}}) and refuses a generation with no rows and no mention in the projection state ({:error, :not_found}).

Pruning the generation a replay abandoned also drops its seed rows and clears it from the projection state, so the next run/1 starts a fresh build.

Examples

iex> AuroraMeter.Events.Replay.prune(0)
{:error, :active}

run(opts \\ [])

@spec run(keyword()) ::
  {:ok, report()}
  | {:ok, :paused, status()}
  | {:error, {:projection_mismatch, map()}}
  | {:error, {:already_running, status()}}
  | {:error, term()}

Rebuilds the projection into a new generation and, by default, activates it.

Options:

  • :batch_size: events per committed batch, default 5000.
  • :compare: :require_match (default) refuses to activate when the rebuilt generation differs from the live one and returns {:error, {:projection_mismatch, summary}}, leaving the built generation for inspection. :report returns the differences and activates anyway, which is the mode for repairing a projection already known to be wrong.
  • :activate: default true. false builds and compares only.
  • :resume: default true. false refuses when a building generation already exists rather than continuing it.
  • :generation: refuse unless the building generation is this one. For an operator who wants to be sure which build they are resuming.
  • :compare_limit: differences listed, default 1000. The count is always complete.
  • :max_batches: stop cleanly after this many committed batches, leaving the cursor where it is. For a bounded maintenance window.
  • :rehydrate: default true. Re-seats warm events-source counter keys after activation so dashboards move without a restart.
  • :timeout: milliseconds for one statement and one batch transaction, default 60000.

A paused replay returns {:ok, :paused, status} at its next batch boundary with everything in place; AuroraMeter.Checkpoints.resume/1 and another run/1 carry on from the cursor.

status()

@spec status() :: status()

The generations and, when one is in progress, the replay's own checkpoint.

Examples

iex> status = AuroraMeter.Events.Replay.status()
iex> is_integer(status.active_generation)
true