AuroraMeter. Oban
(Aurora Meter v1.0.0)
View Source
Optional Oban workers for Aurora Meter's scheduled operations, and the recommended crontab for them.
This module and everything under it exist only when oban is installed.
Aurora Meter does not depend on Oban: every operation a worker here wraps is
a public function a host can call from any scheduler, or from iex. See
the scheduler map for the direct-call recipes and
for the Aurora Meter Pro half of the inventory.
config :my_app, Oban,
repo: MyApp.Repo,
queues: [aurora_meter: 5],
plugins: [
{Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()},
{Oban.Plugins.Lifeline, rescue_after: :timer.minutes(20)}
]The second plugin is not decoration. A node killed while a worker is running
leaves its job executing and nothing ever moves it, because nothing
observed the death; Lifeline is what clears it. validate!/1 warns when it
is absent, and the scheduler map says how to pick
rescue_after.
What a worker is, and what it is not
A worker here is a perform/1 that calls one operation and maps its result
onto an Oban result. It opens no transaction, takes no lock, holds no state
between runs and reads no clock to decide anything. Every guarantee lives in
the operation.
That is deliberate, and it is what makes running two of them safe. Aurora
Meter does not assume a job runs a single time: a cron plugin can tick twice
across a leader change, a rescued job runs again, and two nodes can both be
told to sweep. What each operation guarantees instead is that a second run
finds the work already done and says so. AuroraMeter.Credits.expire_due/1
re-reads expired_at under the grant row's own FOR UPDATE;
AuroraMeter.Credits.reconcile_holds/1 re-reads status under the hold
row's. Neither uses a lease, a fence or a duration, so neither can be
inverted by a clock that steps backwards.
The unique option on each worker is therefore defence in depth and not the
guarantee. Removing it wastes work; it does not move money.
Its period is finite on every worker here, and that is not decoration
either. :executing is among the uniqueness states, so a job a dead node
left executing deduplicates new enqueues; with period: :infinity it did
so for ever, and one node death stopped that worker permanently
(open-findings.md X486). The periods, and the rule that produced them, are
in the scheduler map.
Availability
cron_entries/1 returns an entry only for a worker whose operation is
compiled into this build. Two of the six workers wrap operations that
Aurora Meter 1.0 adds after this module: until those land, the worker exists
(so a crontab written by hand cannot name a module that is missing), it is
absent from cron_entries/1, and running it by hand cancels the job with
{:cancel, :not_implemented} rather than failing it repeatedly. One of the
two, AuroraMeter.Oban.RecurringGrants, has since had its operation land,
and it began appearing in cron_entries/1 with no edit to the worker at
all, which is what the check is for.
Summary
Types
One crontab entry, in the shape Oban.Plugins.Cron accepts.
Functions
The recommended crontab entries for the workers this build can run.
The queue every Aurora Meter worker declares, core and Pro alike.
The rescue advice validate!/1 would give for this configuration, as a list
of sentences, empty when there is nothing to say.
Checks a host's Oban configuration against what these workers need, and
raises AuroraMeter.Oban.ConfigError listing everything that is wrong.
Types
One crontab entry, in the shape Oban.Plugins.Cron accepts.
Functions
@spec cron_entries(keyword()) :: [cron_entry()]
The recommended crontab entries for the workers this build can run.
Pass the result straight to Oban.Plugins.Cron. An entry appears only when
the operation behind it is compiled into this build and the worker has a
schedule, so a crontab built from this can never name a job that will cancel
itself.
Options:
:include: only these workers. Each is a module or its short name (:credit_expiryforAuroraMeter.Oban.CreditExpiry).:exclude: everything but these, in the same two forms.:schedules: a map of worker (or short name) to cron expression, overriding the default. Naming a worker that has no default schedule here is how an operator deliberately schedules one; nothing else returns it.
Examples
iex> AuroraMeter.Oban.cron_entries()
[
{"*/30 * * * *", AuroraMeter.Oban.CreditExpiry},
{"*/15 * * * *", AuroraMeter.Oban.HoldReconciliation},
{"7 * * * *", AuroraMeter.Oban.RecurringGrants},
{"*/5 * * * *", AuroraMeter.Oban.PlanTransitions},
{"40 3 * * *", AuroraMeter.Oban.Retention}
]
iex> AuroraMeter.Oban.cron_entries(include: [:credit_expiry])
[{"*/30 * * * *", AuroraMeter.Oban.CreditExpiry}]
iex> AuroraMeter.Oban.cron_entries(
...> exclude: [
...> AuroraMeter.Oban.HoldReconciliation,
...> AuroraMeter.Oban.RecurringGrants,
...> AuroraMeter.Oban.PlanTransitions,
...> AuroraMeter.Oban.Retention
...> ],
...> schedules: %{credit_expiry: "0 4 * * *"}
...> )
[{"0 4 * * *", AuroraMeter.Oban.CreditExpiry}]
@spec queue() :: atom()
The queue every Aurora Meter worker declares, core and Pro alike.
One queue, because the two packages' workers are the same kind of work and a host that sizes one has sized both.
Examples
iex> AuroraMeter.Oban.queue()
:aurora_meter
The rescue advice validate!/1 would give for this configuration, as a list
of sentences, empty when there is nothing to say.
Exposed so a host can put it in a health check or a boot banner of its own instead of reading the log.
Examples
iex> AuroraMeter.Oban.rescue_advice(
...> plugins: [{Oban.Plugins.Cron, crontab: [{"*/5 * * * *", AuroraMeter.Oban.PlanTransitions}]}]
...> ) |> length()
1
iex> AuroraMeter.Oban.rescue_advice(
...> plugins: [
...> {Oban.Plugins.Cron, crontab: [{"*/5 * * * *", AuroraMeter.Oban.PlanTransitions}]},
...> Oban.Plugins.Lifeline
...> ]
...> )
[]
@spec validate!(keyword()) :: :ok
Checks a host's Oban configuration against what these workers need, and
raises AuroraMeter.Oban.ConfigError listing everything that is wrong.
It reads configuration, never a running Oban instance, so it has no
boot-ordering trap: call it from your Application.start/2 before the
children list, whether Oban starts before or after Aurora Meter.
def start(_type, _args) do
if Code.ensure_loaded?(AuroraMeter.Oban) do
AuroraMeter.Oban.validate!(otp_app: :my_app)
end
Supervisor.start_link(children, opts)
endOptions: either :config (the host's Oban keyword list) or :otp_app with
an optional :name (default Oban) to look it up, plus an optional
:queue (default :aurora_meter) and an optional :env (default
Mix.env/0 where Mix exists, and nil in a release, where there is no
environment to read).
What it refuses:
:repomissing, or not the repoAuroraMeter.Config.repo/0returns;- the Aurora Meter queue absent from
:queues, or its limit zero; - a
:crontabentry naming anAuroraMeter.Oban.*module this build does not have; - a
:crontabnaming one worker twice, or naming bothAuroraMeter.Oban.CreditExpiryandAuroraMeter.Pro.Credits.Expirer, which would sweep expiry twice for one result; - a
Oban.Plugins.Cron:timezonethatDateTime.now/1cannot resolve, which is the same test Oban itself applies, run early enough to name the key; :testingset to:inlineor:manualoutside the test environment, which silently stops every scheduled job.
What it warns about: no way to rescue an orphaned job
A node killed while one of these workers is running leaves that job
executing for ever, because nothing observes the death. Oban's answer is
Oban.Plugins.Lifeline, and a configuration that schedules Aurora Meter
work without it (or without Oban Pro's DynamicLifeline) accumulates
orphans that no retry, no alert and no operator ever sees:
plugins: [
{Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()},
{Oban.Plugins.Lifeline, rescue_after: :timer.minutes(20)}
]This is a warning, logged once, and never a refusal unless you ask for
one with rescue: :require. Aurora Meter's own workers no longer depend on
it for liveness: every uniqueness period is finite, so a wedged job stops
that worker for at most its period rather than for ever
(open-findings.md X486). What the plugin adds is the cleanup, and
rescue_after is a trade an operator makes rather than one this library can
make for them: shorter than the harm, and longer than your slowest
legitimate run, because Lifeline rescues on elapsed time alone and will
happily rescue a job that is still running.
Pass rescue: :ignore if you rescue orphans some other way and do not want
to be told.
It cannot see a second Oban.Plugins.Cron running under a different Oban
instance name. That case is documented in the scheduler map.
Examples
config = [
repo: MyApp.Repo,
queues: [aurora_meter: 5],
plugins: [{Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()}]
]
:ok = AuroraMeter.Oban.validate!(config: config)