AuroraMeter. Oban. HoldReconciliation
(Aurora Meter v1.0.0)
View Source
Asks the configured AuroraMeter.Credits.HoldReconciler about holds that
are still open past a cutoff, and applies what it says.
One call to AuroraMeter.Credits.reconcile_holds/1 and nothing else.
config :my_app, Oban,
plugins: [{Oban.Plugins.Cron, crontab: [
{"*/15 * * * *", AuroraMeter.Oban.HoldReconciliation}
]}]With no :credits_hold_reconciler configured this worker keeps every
hold, which is the default and is why scheduling it changes nothing until
a host writes a policy. Read
AuroraMeter.Credits.HoldReconciler before configuring one: age is not
evidence that work was abandoned.
Job arguments
| Key | Default | Meaning |
|---|---|---|
"older_than_seconds" | 3600 | Only holds open longer than this are examined. |
"limit" | 200 | The most holds one batch examines. |
"max_batches" | 10 | Batches one job runs before returning. |
"reference_prefix" | none | Narrow to one kind of work. |
"tenant" | none | Sweep one tenant. |
%{"older_than_seconds" => 86_400, "reference_prefix" => "job:"}
|> AuroraMeter.Oban.HoldReconciliation.new()
|> Oban.insert()The cursor is never a job argument (L05c-1). It lives in
aurora_meter_checkpoints under "hold_reconciliation:global", so two jobs
for this worker read the same position rather than each carrying its own
stale copy.
Pausing
AuroraMeter.Operations.pause("hold_reconciliation:global")The next batch boundary cancels the job with {:cancel, :paused}, leaving
the cursor where the last batch left it.
The cutoff is pinned for the length of a scan
older_than_seconds selects the candidate set, and the set moves as time
passes. A scan spanning several jobs stores the cutoff it started with in
its checkpoint beside the cursor, so a resumed cursor points into the set it
came from. When the scan completes, the cursor is cleared and the next run
takes a fresh cutoff.
The cutoff is a duration, and it is an hours-scale one
older_than_seconds is the one duration in this worker. It selects
candidates and decides nothing: having passed the cutoff makes a hold a
hold the host is asked about, never a hold that is released. The default is
an hour and the smallest value that makes sense is minutes, which is far
outside the sub-second range in which a shared clock cannot be trusted to
order two instants (AuroraMeter.Clock).
The cutoff is measured with AuroraMeter.Clock.now/0 rather than
db_now/0, because the other side of the comparison is a hold row's
inserted_at, and until the ledger's rows are stamped by the database that
is a node's wall clock. Both sides of a time comparison come from the same
clock, and which clock that is follows from where the other side came from.
Running it twice
Harmless, and the operation's doing rather than this worker's. Two nodes
sweeping at the same instant both list a hold and both ask the host about
it, and at most one terminal transition happens: the decision is applied
through AuroraMeter.Credits.settle/3 or AuroraMeter.Credits.release/2,
which re-read
status = 'pending' under the hold row's own FOR UPDATE. The loser is
told :already_closed and counted. There is no lease and no fence, so there
is nothing a clock could invert.
A host callback that raises, hangs or answers nonsense keeps the hold, and so does a run this worker is killed in the middle of.
Without Oban, call AuroraMeter.Credits.reconcile_holds/1 from whatever
scheduler you do have. See the scheduler map.
Summary
Functions
The checkpoint name this worker pauses and resumes under.
Reconciles pending holds in bounded batches and returns {:ok, report}.
Functions
@spec operation() :: String.t()
The checkpoint name this worker pauses and resumes under.
@spec perform(Oban.Job.t()) :: {:ok, term()} | {:error, term()} | {:cancel, term()}
Reconciles pending holds in bounded batches and returns {:ok, report}.
{:cancel, :paused} when the operation is paused. {:error, reason} when
the listing itself fails, which fails the job and lets Oban retry it. It
never raises on an operation error and never matches on the report's shape:
a later release may add keys to it.
Examples
{:ok, report} =
AuroraMeter.Oban.HoldReconciliation.perform(%Oban.Job{
args: %{"older_than_seconds" => 3600}
})