AuroraMeter. Storage behaviour
(Aurora Meter v1.0.0)
View Source
Behaviour for persisting Aurora Meter state, plus dispatch helpers that route to
the configured adapter (AuroraMeter.Config.storage/0, default
AuroraMeter.Storage.Ecto).
Only a Postgres/Ecto adapter ships in v1. Writing another one is supported,
and Storage adapters is the guide: it gives a
copyable minimal adapter, and AuroraMeter.StorageCase is the conformance
suite that proves one correct.
Capabilities
Every callback here is required, including the durable-event ones. An adapter
that cannot do durable work does not omit them: it declares what it supports
from capabilities/0 and the dispatcher answers
{:error, {:unsupported, operation}} without calling the adapter at all. The
difference matters because "this adapter cannot record durable events" is an
answer a caller can handle, and a FunctionClauseError from a missing
callback is not.
Summary
Types
A durable operation an adapter may support.
One correction to record: what the caller stated, and nothing it inherits.
A counter delta to add: value = value + delta.
A counter snapshot to persist. feature may be an atom or string.
A total as returned after adding deltas. feature is a string.
One event to record, already validated and canonicalised by
AuroraMeter.Events.Canonical.
A raw usage event to persist, as the legacy durable-track path writes them.
A day-bucket delta to add.
A day bucket as read back: %{date: Date.t(), value: integer()}.
A day-bucket snapshot to persist.
A day-bucket total as returned after adding deltas.
What projection_state/0 reports and begin_projection_generation/0
returns.
A projection total, as write_projection_totals/2 takes them.
What one recorded event produced: the persisted fact and whether it was new.
Callbacks
Makes generation the one load_event_total/3 reads.
Names the contract of at most limit subscriptions that have no
plan_version, and reports what it did.
Announces a new building generation and returns the state it announced.
The durable operations this adapter supports.
Subtracts up to limit seed rows from the generation they seeded and deletes
them, in one transaction, returning how many were consumed.
Every stored snapshot for plan_id, or for every plan when given :all.
One bounded page of subscriptions, in keyset order.
Reads one recorded event by its caller identity.
Reads the projected total for one feature and period, in the active generation.
Reads the projection state without locking anything.
Inserts one plan version snapshot, or does nothing if (plan_id, version) is
already stored.
Records one correction, and optionally its replacement, in one transaction.
Records a batch of events in one transaction, with their projection deltas and the configured outbox's intent.
Reads one bounded page of events after cursor, ordered by seq.
Adds projection deltas to generation, creating rows that are absent.
Functions
Makes a generation the one reads see. See activate_projection/1.
Adds deltas to counters (value = value + delta, inserting at delta when
the row is new) and returns the resulting totals. This is what makes
cluster-wide counting correct: each node writes only what it added.
Adds deltas to day buckets and returns the resulting totals. See add_counters/1.
Names one batch of legacy subscriptions. See assign_legacy_plan_versions/1.
Announces a building generation. See begin_projection_generation/0.
The durable operations the configured adapter supports.
Consumes a bounded slice of a seed generation. See drain_projection_seed/2.
Atomically applies counter and history deltas once for batch_id.
Fetches a tenant's subscription straight from storage (uncached), or nil.
Appends raw usage events (durable mode / audit).
Reads stored plan version snapshots. See list_plan_versions/1.
One bounded page of subscriptions, keyset by tenant_key.
Loads a single flushed counter value, or nil if absent.
Reads one recorded event by its caller identity. See load_event/2.
Reads the active generation's projected total. See load_event_total/3.
Loads a single flushed day-bucket value, or nil if absent.
Loads the flushed day buckets for a feature between two dates (inclusive), oldest first.
Reads the projection state. See projection_state/0.
Stores one plan version snapshot. See put_plan_version/1.
Inserts or updates a tenant's subscription (upsert on tenant_key) and evicts
it from the subscription cache on every node.
Records one correction (and optionally its replacement). See record_correction/2.
Records a batch of events in one transaction. See record_events/2.
Returns all counter snapshots for a period (used by Pro rollups).
Reads one bounded page of events after cursor. See stream_events/2.
Whether the configured adapter supports capability.
Sets counter snapshots to absolute values by {tenant_key, feature, period_start}. For backfills and test fixtures; the flusher uses
add_counters/1 so that nodes add up instead of overwriting one another.
Upserts day-bucket snapshots (absolute values) by {tenant_key, feature, date}.
Adds projection deltas to a generation. See write_projection_totals/2.
Types
@type capability() ::
:durable_events
| :corrections
| :projection_generations
| :event_streaming
| :plan_versions
A durable operation an adapter may support.
:durable_events covers record_events/2, load_event/2 and
load_event_total/3; :corrections covers record_correction/2 (build
unit 03e); :projection_generations covers begin_projection_generation/0,
projection_state/0, write_projection_totals/2,
drain_projection_seed/2 and activate_projection/1;
:event_streaming covers stream_events/2.
@type correction_entry() :: %{ tenant_key: String.t(), event_id: String.t(), original_event_id: String.t(), quantity: pos_integer() | :remaining, metadata: map() }
One correction to record: what the caller stated, and nothing it inherits.
A correction's feature, occurrence instant, period, plan attribution and
dimensions are the original's, read inside the transaction that holds it
under lock, so they cannot drift from it (L-03e-2). quantity is the
magnitude of the reduction, a positive integer, or :remaining for
AuroraMeter.replace/4, whose magnitude is original.quantity less the
corrections already committed and is therefore not known until then.
There is no payload_hash here for the same reason: the canonical tuple
covers fields that come from the original, so the adapter computes it with
AuroraMeter.Events.Canonical.correction_hash/3 once it has read one.
@type counter_delta() :: %{ tenant_key: String.t(), feature: atom() | String.t(), period_start: DateTime.t(), delta: integer() }
A counter delta to add: value = value + delta.
@type counter_row() :: %{ tenant_key: String.t(), feature: atom() | String.t(), period_start: DateTime.t(), value: integer() }
A counter snapshot to persist. feature may be an atom or string.
@type counter_total() :: %{ tenant_key: String.t(), feature: String.t(), period_start: DateTime.t(), value: integer() }
A total as returned after adding deltas. feature is a string.
@type event_entry() :: %{ :tenant_key => String.t(), :event_id => String.t(), :feature => String.t(), :quantity => pos_integer(), :kind => String.t(), :original_event_id => String.t() | nil, :occurred_at => DateTime.t(), :period_start => DateTime.t(), :period_source => String.t(), :attribution => String.t(), :dimensions => map(), :metadata => map(), :payload_hash => binary(), optional(:plan_id) => String.t() | nil, optional(:plan_version) => String.t() | nil }
One event to record, already validated and canonicalised by
AuroraMeter.Events.Canonical.
feature and kind are strings because that is how the columns hold them,
and an adapter must not have to know which atoms exist.
@type event_row() :: %{ :tenant_key => String.t(), :feature => atom() | String.t(), optional(:quantity) => integer(), optional(:metadata) => map(), optional(:period_start) => DateTime.t() | nil, optional(:period_source) => atom() | String.t() | nil }
A raw usage event to persist, as the legacy durable-track path writes them.
period_start and period_source are the period AuroraMeter.track/4
already resolved for the counter it bumped, passed down rather than looked up
again: a second resolution could land on the other side of a period boundary
from the increment the row accompanies. They are optional so that a 0.4.x
caller of AuroraMeter.Storage.insert_events/1 keeps working, in which case
the row carries no period.
@type history_delta() :: %{ tenant_key: String.t(), feature: atom() | String.t(), date: Date.t(), delta: integer() }
A day-bucket delta to add.
A day bucket as read back: %{date: Date.t(), value: integer()}.
@type history_row() :: %{ tenant_key: String.t(), feature: atom() | String.t(), date: Date.t(), value: integer() }
A day-bucket snapshot to persist.
@type history_total() :: %{ tenant_key: String.t(), feature: String.t(), date: Date.t(), value: integer() }
A day-bucket total as returned after adding deltas.
@type projection_state() :: %{ active_generation: non_neg_integer(), building_generation: integer() | nil, previous_generation: non_neg_integer() | nil, seed_generation: integer() | nil, watermark: non_neg_integer() | nil }
What projection_state/0 reports and begin_projection_generation/0
returns.
building_generation is present while a replay is building one;
seed_generation names the frozen copy of the active generation that the
announcement took, and watermark the seq the scan is bounded by. All
three are absent between replays.
@type projection_total() :: %{ tenant_key: String.t(), feature: String.t(), period_start: DateTime.t(), quantity: non_neg_integer(), events: non_neg_integer() }
A projection total, as write_projection_totals/2 takes them.
@type record_outcome() :: {AuroraMeter.Event.t(), :inserted | :duplicate}
What one recorded event produced: the persisted fact and whether it was new.
Callbacks
@callback activate_projection(non_neg_integer()) :: :ok | {:error, term()}
Makes generation the one load_event_total/3 reads.
@callback add_counters([counter_delta()]) :: {:ok, [counter_total()]}
@callback add_history([history_delta()]) :: {:ok, [history_total()]}
@callback assign_legacy_plan_versions(pos_integer()) :: {:ok, %{assigned: non_neg_integer(), orphans: non_neg_integer()}} | {:error, term()} | {:error, {:unsupported, capability()}}
Names the contract of at most limit subscriptions that have no
plan_version, and reports what it did.
One bounded transaction. Rows are taken with FOR UPDATE SKIP LOCKED, so two
nodes running AuroraMeter.Plans.register!/0 at the same time take disjoint
batches rather than queueing. assigned is how many rows this batch named and
orphans how many of those carry a plan_id with no registered version, which
get a version and a NULL fingerprint: retiring a plan id from code is ordinary
and refusing the upgrade for every customer who still holds one would make the
upgrade unrunnable for exactly the installs with the longest histories.
assigned: 0 means there is nothing left to do and the caller stops.
Not one of the two callbacks api-change-map.md 1.2 lists. It is here
rather than as raw SQL inside AuroraMeter.Plans so that the whole of this
feature sits behind one capability, and an adapter that cannot store snapshots
is not asked to run a statement against a table it does not have (finding
X286).
Announces a new building generation and returns the state it announced.
One short transaction that takes the projection row FOR UPDATE, so it is
granted only once every record transaction holding FOR SHARE on that row
has committed or rolled back. In that transaction it reads the watermark
max(seq), copies the active generation's rows into both the new building
generation and a frozen seed generation, and publishes the building
generation so every later record writes both.
Idempotent: when a building generation already exists it is returned with
resumed: true and nothing is copied.
@callback capabilities() :: [capability()]
The durable operations this adapter supports.
An adapter that returns [] still defines every callback below; the
dispatchers refuse the call on its behalf.
@callback drain_projection_seed(integer(), pos_integer()) :: {:ok, non_neg_integer()} | {:error, term()}
Subtracts up to limit seed rows from the generation they seeded and deletes
them, in one transaction, returning how many were consumed.
The seed row's own existence is the cursor: draining is finished when none is left, so an interrupted drain needs no checkpoint and can never subtract a row twice.
@callback flush_batch(Ecto.UUID.t(), [counter_delta()], [history_delta()]) :: {:ok, %{counters: [counter_total()], history: [history_total()]}} | {:error, term()}
@callback get_subscription(String.t()) :: AuroraMeter.Schema.Subscription.t() | nil
@callback insert_events([event_row()]) :: :ok
@callback list_plan_versions(String.t() | :all) :: [AuroraMeter.Schema.PlanVersion.t()] | {:error, {:unsupported, capability()}}
Every stored snapshot for plan_id, or for every plan when given :all.
Ordered by (plan_id, version) so two nodes comparing fingerprints compare
them in the same order and report the same list.
@callback list_subscriptions(cursor :: String.t() | nil, opts :: keyword()) :: {[AuroraMeter.Schema.Subscription.t()], next_cursor :: String.t() | nil}
One bounded page of subscriptions, in keyset order.
cursor is the opaque value the previous page ended on, or nil for the
first page; the page returned is strictly after it. The second element of the
return is the cursor to pass next, or nil when that page was the last one.
Options:
:limit(default 100);:status_in: a list of status strings; omitted means every status;:order::tenant_key(default) or:scheduled_effective_at, which walks pending plan transitions in the order they come due;:transition_state,:transition_confirm: exact matches on the transition mirror columns, which is how Aurora Meter Pro finds the provider-confirmed work it owns without querying a core table directly;:scheduled_before: aDateTime, inclusive.
Keyset, not offset, because the caller is a worker that walks every subscription while other processes insert and delete them. An offset page silently skips a row when an earlier one is removed under it, and a skipped subscription is usage nobody bills.
@callback load_counter(String.t(), atom() | String.t(), DateTime.t()) :: integer() | nil
@callback load_event(String.t(), String.t()) :: {:ok, AuroraMeter.Event.t()} | {:error, :not_found | {:unsupported, capability()}}
Reads one recorded event by its caller identity.
@callback load_event_total(String.t(), atom() | String.t(), DateTime.t()) :: {:ok, %{quantity: non_neg_integer(), events: non_neg_integer()}} | {:error, {:unsupported, capability()}}
Reads the projected total for one feature and period, in the active generation.
@callback projection_state() :: {:ok, projection_state()} | {:error, term()}
Reads the projection state without locking anything.
@callback put_plan_version(map()) :: {:ok, AuroraMeter.Schema.PlanVersion.t()} | {:error, term()} | {:error, {:unsupported, capability()}}
Inserts one plan version snapshot, or does nothing if (plan_id, version) is
already stored.
Never an update. A snapshot is the commercial content a customer was sold and
it is immutable; a differing compiled definition is
AuroraMeter.PlanVersionConflictError, not a write.
@callback put_subscription(map()) :: {:ok, AuroraMeter.Schema.Subscription.t()} | {:error, Ecto.Changeset.t()}
@callback record_correction( correction_entry(), keyword() ) :: {:ok, [record_outcome()]} | {:error, {:conflict, non_neg_integer(), AuroraMeter.Event.t()}} | {:error, {:invalid, [{atom(), atom()}]}} | {:error, {:not_found, :original}} | {:error, term()}
Records one correction, and optionally its replacement, in one transaction.
A correction is a new immutable row reducing the effective quantity of an
existing event in the same tenant; no historical row is ever updated. The
cumulative magnitude of the corrections of one original may never exceed that
original's quantity (I09), which an adapter enforces by serialising every
corrector of that original against one another. The Ecto adapter does it with
SELECT ... FOR UPDATE on the original row.
The order of the two checks is load bearing and an adapter must keep it: the
duplicate check comes before the bound check. A retry of a correction has
its own committed row inside the cumulative sum, so evaluating the bound
first makes every retry of a correct correction fail with
exceeds_original, which is the difference between an idempotent financial
operation and one that cannot be retried at all.
Results are [{correction, outcome}], or [{correction, outcome}, {replacement, outcome}] when opts[:replacement] carries a usage entry to
insert in the same transaction (AuroraMeter.replace/4).
Refusals return {:error, {:invalid, _}} or {:error, {:not_found, :original}} without rolling back, so a host transaction that wrapped the
call keeps its own writes. Conflicts roll back, because by then a write has
been attempted and the caller's intent is unsatisfiable.
Options: :timeout, :outbox and :replacement.
@callback record_events( [event_entry()], keyword() ) :: {:ok, [record_outcome()]} | {:error, {:conflict, non_neg_integer(), AuroraMeter.Event.t()}} | {:error, term()}
Records a batch of events in one transaction, with their projection deltas and the configured outbox's intent.
The three are one commit or none of them (L-03b-1). Results are returned in
the caller's input order. An entry whose (tenant_key, event_id) already
exists with an equal payload_hash is :duplicate and contributes no totals
delta and no outbox item; one with a different hash rolls the whole batch
back with {:error, {:conflict, index, existing}}.
Options: :timeout (milliseconds for the transaction and every statement in
it) and :outbox (the AuroraMeter.Events.Outbox module to invoke).
@callback stream_counters(DateTime.t()) :: [AuroraMeter.Schema.Counter.t()]
@callback stream_events( non_neg_integer(), keyword() ) :: {:ok, [AuroraMeter.Event.t()]} | {:error, {:unsupported, capability()}}
Reads one bounded page of events after cursor, ordered by seq.
seq and not id: event ids are random v4 UUIDs, so a keyset scan ordered
by id can silently miss a row committed by a transaction that started
earlier (open-findings.md L20).
Options: :limit, :tenant, :feature, :from and :to (on
occurred_at, half-open).
@callback upsert_counters([counter_row()]) :: :ok
@callback upsert_history([history_row()]) :: :ok
@callback write_projection_totals(non_neg_integer(), [projection_total()]) :: :ok | {:error, term()}
Adds projection deltas to generation, creating rows that are absent.
Adds rather than sets, because a record that commits while a generation is
being built writes its own delta there too, and a replay's batch must not
overwrite it. Entries are applied in one statement, sorted by
{tenant_key, feature, period_start} so two writers cannot deadlock.
Call it inside the caller's transaction: a replay commits the totals and its checkpoint together, which is the whole of its resumability.
Functions
@spec activate_projection(non_neg_integer()) :: :ok | {:error, term()}
Makes a generation the one reads see. See activate_projection/1.
@spec add_counters([counter_delta()]) :: {:ok, [counter_total()]}
Adds deltas to counters (value = value + delta, inserting at delta when
the row is new) and returns the resulting totals. This is what makes
cluster-wide counting correct: each node writes only what it added.
@spec add_history([history_delta()]) :: {:ok, [history_total()]}
Adds deltas to day buckets and returns the resulting totals. See add_counters/1.
@spec assign_legacy_plan_versions(pos_integer()) :: {:ok, %{assigned: non_neg_integer(), orphans: non_neg_integer()}} | {:error, term()} | {:error, {:unsupported, capability()}}
Names one batch of legacy subscriptions. See assign_legacy_plan_versions/1.
Announces a building generation. See begin_projection_generation/0.
@spec capabilities() :: [capability()]
The durable operations the configured adapter supports.
Examples
iex> :durable_events in AuroraMeter.Storage.capabilities()
true
@spec drain_projection_seed(integer(), pos_integer()) :: {:ok, non_neg_integer()} | {:error, term()}
Consumes a bounded slice of a seed generation. See drain_projection_seed/2.
@spec flush_batch(Ecto.UUID.t(), [counter_delta()], [history_delta()]) :: {:ok, %{counters: [counter_total()], history: [history_total()]}} | {:error, term()}
Atomically applies counter and history deltas once for batch_id.
Adapters must persist the receipt and both sets of deltas in one transaction. A retry returns the current totals without applying either delta again.
Examples
{:ok, %{counters: [], history: []}} = AuroraMeter.Storage.flush_batch(Ecto.UUID.generate(), [], [])
@spec get_subscription(String.t()) :: AuroraMeter.Schema.Subscription.t() | nil
Fetches a tenant's subscription straight from storage (uncached), or nil.
@spec insert_events([event_row()]) :: :ok
Appends raw usage events (durable mode / audit).
@spec list_plan_versions(String.t() | :all) :: [AuroraMeter.Schema.PlanVersion.t()] | {:error, {:unsupported, capability()}}
Reads stored plan version snapshots. See list_plan_versions/1.
@spec list_subscriptions( String.t() | nil, keyword() ) :: {[AuroraMeter.Schema.Subscription.t()], String.t() | nil}
One bounded page of subscriptions, keyset by tenant_key.
{rows, next_cursor}; pass next_cursor back for the following page and
stop when it is nil. Options: :limit (default 100), :status_in.
{page, cursor} = AuroraMeter.Storage.list_subscriptions(nil, limit: 200)
{next, cursor} = AuroraMeter.Storage.list_subscriptions(cursor, limit: 200)This is the supported way for Aurora Meter Pro to walk subscriptions; it
replaces the direct AuroraMeter.Schema.Subscription query the usage
reporter used to run (free-pro-boundary.md rule 1).
@spec load_counter(String.t(), atom() | String.t(), DateTime.t()) :: integer() | nil
Loads a single flushed counter value, or nil if absent.
@spec load_event(String.t(), String.t()) :: {:ok, AuroraMeter.Event.t()} | {:error, :not_found | {:unsupported, capability()}}
Reads one recorded event by its caller identity. See load_event/2.
@spec load_event_total(String.t(), atom() | String.t(), DateTime.t()) :: {:ok, %{quantity: non_neg_integer(), events: non_neg_integer()}} | {:error, {:unsupported, capability()}}
Reads the active generation's projected total. See load_event_total/3.
Loads a single flushed day-bucket value, or nil if absent.
@spec load_history_range(String.t(), atom() | String.t(), Date.t(), Date.t()) :: [ history_point() ]
Loads the flushed day buckets for a feature between two dates (inclusive), oldest first.
@spec projection_state() :: {:ok, projection_state()} | {:error, term()}
Reads the projection state. See projection_state/0.
@spec put_plan_version(map()) :: {:ok, AuroraMeter.Schema.PlanVersion.t()} | {:error, term()} | {:error, {:unsupported, capability()}}
Stores one plan version snapshot. See put_plan_version/1.
@spec put_subscription(map()) :: {:ok, AuroraMeter.Schema.Subscription.t()} | {:error, Ecto.Changeset.t()}
Inserts or updates a tenant's subscription (upsert on tenant_key) and evicts
it from the subscription cache on every node.
It reacts to a provider-driven plan change
A customer who changes plan in the billing provider's own portal reaches
Aurora Meter here, not through AuroraMeter.Subscriptions. So the reaction
to that lives here, where every provider path already passes, rather than in
a call the provider integration could forget to make. When the tenant has a
pending plan transition and this write changes the plan:
- writing a status that is not in
AuroraMeter.Schema.Subscription.entitled_statuses/0cancels it withdetail.reason = "subscription_not_entitled"; - writing exactly the scheduled
{plan_id, plan_version}applies it early withdetail.reason = "provider_applied_early"; - writing any other pair cancels it with
detail.reason = "provider_override"and records what was observed.
A write that changes nothing about the plan leaves the transition pending. See Plans for the full precedence table and which rows belong to Aurora Meter Pro.
This is a new side effect on an existing public function (1.0.0-rc.1). A
host calling put_subscription/1 directly with a changed plan while a
transition is pending will now see that transition settled.
A tenant with no pending transition pays one uncached read and opens no transaction, which is every call on an ordinary installation.
@spec record_correction( correction_entry(), keyword() ) :: {:ok, [record_outcome()]} | {:error, {:conflict, non_neg_integer(), AuroraMeter.Event.t()}} | {:error, {:invalid, [{atom(), atom()}]}} | {:error, {:not_found, :original}} | {:error, term()}
Records one correction (and optionally its replacement). See record_correction/2.
@spec record_events( [event_entry()], keyword() ) :: {:ok, [record_outcome()]} | {:error, {:conflict, non_neg_integer(), AuroraMeter.Event.t()}} | {:error, term()}
Records a batch of events in one transaction. See record_events/2.
@spec stream_counters(DateTime.t()) :: [AuroraMeter.Schema.Counter.t()]
Returns all counter snapshots for a period (used by Pro rollups).
@spec stream_events( non_neg_integer(), keyword() ) :: {:ok, [AuroraMeter.Event.t()]} | {:error, {:unsupported, capability()}}
Reads one bounded page of events after cursor. See stream_events/2.
@spec supports?(capability()) :: boolean()
Whether the configured adapter supports capability.
Examples
iex> AuroraMeter.Storage.supports?(:corrections)
true
@spec upsert_counters([counter_row()]) :: :ok
Sets counter snapshots to absolute values by {tenant_key, feature, period_start}. For backfills and test fixtures; the flusher uses
add_counters/1 so that nodes add up instead of overwriting one another.
@spec upsert_history([history_row()]) :: :ok
Upserts day-bucket snapshots (absolute values) by {tenant_key, feature, date}.
@spec write_projection_totals(non_neg_integer(), [projection_total()]) :: :ok | {:error, term()}
Adds projection deltas to a generation. See write_projection_totals/2.