AuroraMeter.Oban (Aurora Meter v1.0.0)

View Source

Optional Oban workers for Aurora Meter's scheduled operations, and the recommended crontab for them.

This module and everything under it exist only when oban is installed. Aurora Meter does not depend on Oban: every operation a worker here wraps is a public function a host can call from any scheduler, or from iex. See the scheduler map for the direct-call recipes and for the Aurora Meter Pro half of the inventory.

config :my_app, Oban,
  repo: MyApp.Repo,
  queues: [aurora_meter: 5],
  plugins: [
    {Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()},
    {Oban.Plugins.Lifeline, rescue_after: :timer.minutes(20)}
  ]

The second plugin is not decoration. A node killed while a worker is running leaves its job executing and nothing ever moves it, because nothing observed the death; Lifeline is what clears it. validate!/1 warns when it is absent, and the scheduler map says how to pick rescue_after.

What a worker is, and what it is not

A worker here is a perform/1 that calls one operation and maps its result onto an Oban result. It opens no transaction, takes no lock, holds no state between runs and reads no clock to decide anything. Every guarantee lives in the operation.

That is deliberate, and it is what makes running two of them safe. Aurora Meter does not assume a job runs a single time: a cron plugin can tick twice across a leader change, a rescued job runs again, and two nodes can both be told to sweep. What each operation guarantees instead is that a second run finds the work already done and says so. AuroraMeter.Credits.expire_due/1 re-reads expired_at under the grant row's own FOR UPDATE; AuroraMeter.Credits.reconcile_holds/1 re-reads status under the hold row's. Neither uses a lease, a fence or a duration, so neither can be inverted by a clock that steps backwards.

The unique option on each worker is therefore defence in depth and not the guarantee. Removing it wastes work; it does not move money.

Its period is finite on every worker here, and that is not decoration either. :executing is among the uniqueness states, so a job a dead node left executing deduplicates new enqueues; with period: :infinity it did so for ever, and one node death stopped that worker permanently (open-findings.md X486). The periods, and the rule that produced them, are in the scheduler map.

Availability

cron_entries/1 returns an entry only for a worker whose operation is compiled into this build. Two of the six workers wrap operations that Aurora Meter 1.0 adds after this module: until those land, the worker exists (so a crontab written by hand cannot name a module that is missing), it is absent from cron_entries/1, and running it by hand cancels the job with {:cancel, :not_implemented} rather than failing it repeatedly. One of the two, AuroraMeter.Oban.RecurringGrants, has since had its operation land, and it began appearing in cron_entries/1 with no edit to the worker at all, which is what the check is for.

Summary

Types

One crontab entry, in the shape Oban.Plugins.Cron accepts.

Functions

The recommended crontab entries for the workers this build can run.

The queue every Aurora Meter worker declares, core and Pro alike.

The rescue advice validate!/1 would give for this configuration, as a list of sentences, empty when there is nothing to say.

Checks a host's Oban configuration against what these workers need, and raises AuroraMeter.Oban.ConfigError listing everything that is wrong.

Types

cron_entry()

@type cron_entry() :: {String.t(), module()}

One crontab entry, in the shape Oban.Plugins.Cron accepts.

Functions

cron_entries(opts \\ [])

@spec cron_entries(keyword()) :: [cron_entry()]

The recommended crontab entries for the workers this build can run.

Pass the result straight to Oban.Plugins.Cron. An entry appears only when the operation behind it is compiled into this build and the worker has a schedule, so a crontab built from this can never name a job that will cancel itself.

Options:

  • :include: only these workers. Each is a module or its short name (:credit_expiry for AuroraMeter.Oban.CreditExpiry).
  • :exclude: everything but these, in the same two forms.
  • :schedules: a map of worker (or short name) to cron expression, overriding the default. Naming a worker that has no default schedule here is how an operator deliberately schedules one; nothing else returns it.

Examples

iex> AuroraMeter.Oban.cron_entries()
[
  {"*/30 * * * *", AuroraMeter.Oban.CreditExpiry},
  {"*/15 * * * *", AuroraMeter.Oban.HoldReconciliation},
  {"7 * * * *", AuroraMeter.Oban.RecurringGrants},
  {"*/5 * * * *", AuroraMeter.Oban.PlanTransitions},
  {"40 3 * * *", AuroraMeter.Oban.Retention}
]

iex> AuroraMeter.Oban.cron_entries(include: [:credit_expiry])
[{"*/30 * * * *", AuroraMeter.Oban.CreditExpiry}]

iex> AuroraMeter.Oban.cron_entries(
...>   exclude: [
...>     AuroraMeter.Oban.HoldReconciliation,
...>     AuroraMeter.Oban.RecurringGrants,
...>     AuroraMeter.Oban.PlanTransitions,
...>     AuroraMeter.Oban.Retention
...>   ],
...>   schedules: %{credit_expiry: "0 4 * * *"}
...> )
[{"0 4 * * *", AuroraMeter.Oban.CreditExpiry}]

queue()

@spec queue() :: atom()

The queue every Aurora Meter worker declares, core and Pro alike.

One queue, because the two packages' workers are the same kind of work and a host that sizes one has sized both.

Examples

iex> AuroraMeter.Oban.queue()
:aurora_meter

rescue_advice(config)

@spec rescue_advice(keyword()) :: [String.t()]

The rescue advice validate!/1 would give for this configuration, as a list of sentences, empty when there is nothing to say.

Exposed so a host can put it in a health check or a boot banner of its own instead of reading the log.

Examples

iex> AuroraMeter.Oban.rescue_advice(
...>   plugins: [{Oban.Plugins.Cron, crontab: [{"*/5 * * * *", AuroraMeter.Oban.PlanTransitions}]}]
...> ) |> length()
1

iex> AuroraMeter.Oban.rescue_advice(
...>   plugins: [
...>     {Oban.Plugins.Cron, crontab: [{"*/5 * * * *", AuroraMeter.Oban.PlanTransitions}]},
...>     Oban.Plugins.Lifeline
...>   ]
...> )
[]

validate!(opts \\ [])

@spec validate!(keyword()) :: :ok

Checks a host's Oban configuration against what these workers need, and raises AuroraMeter.Oban.ConfigError listing everything that is wrong.

It reads configuration, never a running Oban instance, so it has no boot-ordering trap: call it from your Application.start/2 before the children list, whether Oban starts before or after Aurora Meter.

def start(_type, _args) do
  if Code.ensure_loaded?(AuroraMeter.Oban) do
    AuroraMeter.Oban.validate!(otp_app: :my_app)
  end

  Supervisor.start_link(children, opts)
end

Options: either :config (the host's Oban keyword list) or :otp_app with an optional :name (default Oban) to look it up, plus an optional :queue (default :aurora_meter) and an optional :env (default Mix.env/0 where Mix exists, and nil in a release, where there is no environment to read).

What it refuses:

  • :repo missing, or not the repo AuroraMeter.Config.repo/0 returns;
  • the Aurora Meter queue absent from :queues, or its limit zero;
  • a :crontab entry naming an AuroraMeter.Oban.* module this build does not have;
  • a :crontab naming one worker twice, or naming both AuroraMeter.Oban.CreditExpiry and AuroraMeter.Pro.Credits.Expirer, which would sweep expiry twice for one result;
  • a Oban.Plugins.Cron :timezone that DateTime.now/1 cannot resolve, which is the same test Oban itself applies, run early enough to name the key;
  • :testing set to :inline or :manual outside the test environment, which silently stops every scheduled job.

What it warns about: no way to rescue an orphaned job

A node killed while one of these workers is running leaves that job executing for ever, because nothing observes the death. Oban's answer is Oban.Plugins.Lifeline, and a configuration that schedules Aurora Meter work without it (or without Oban Pro's DynamicLifeline) accumulates orphans that no retry, no alert and no operator ever sees:

plugins: [
  {Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()},
  {Oban.Plugins.Lifeline, rescue_after: :timer.minutes(20)}
]

This is a warning, logged once, and never a refusal unless you ask for one with rescue: :require. Aurora Meter's own workers no longer depend on it for liveness: every uniqueness period is finite, so a wedged job stops that worker for at most its period rather than for ever (open-findings.md X486). What the plugin adds is the cleanup, and rescue_after is a trade an operator makes rather than one this library can make for them: shorter than the harm, and longer than your slowest legitimate run, because Lifeline rescues on elapsed time alone and will happily rescue a job that is still running.

Pass rescue: :ignore if you rescue orphans some other way and do not want to be told.

It cannot see a second Oban.Plugins.Cron running under a different Oban instance name. That case is documented in the scheduler map.

Examples

config = [
  repo: MyApp.Repo,
  queues: [aurora_meter: 5],
  plugins: [{Oban.Plugins.Cron, crontab: AuroraMeter.Oban.cron_entries()}]
]

:ok = AuroraMeter.Oban.validate!(config: config)