Service levels and limits

This page states the service levels a dynamo-core deployment is built and tested to, the limits they were measured at, and the metrics and alerts that watch them. The load test (make test-load) measures every one of them on a local deployment. They are design targets, not commitments: the core does not enforce them, and they are not an offer of service to anyone. What a status page can show from the same metrics is on status page.

Service levels

DeployableAvailabilityAdmission latency (p95)Settlement lag
dynamo-core, one tenant99.9 % of admissions answered without a server error250 ms or less15 minutes or less
dynamo-core, many tenants (the router)99.9 % of admissions answered without a server error250 ms or less15 minutes or less

Tested limits

make test-load runs two legs, one tenant and many tenants, against a deployment booted from the repository, in Control Mode:

ProfileOne tenantMany tenants
Reduced: part of the full gate20 budgets × 5 streams, 100 admissions per second for 20 seconds3 tenants × 4 streams, 30 admissions per second for 15 seconds
Full: the soak, started by hand (LOAD_PROFILE=full)200 budgets × 10 streams, 300 admissions per second for 30 minutes20 tenants × 10 streams, 100 admissions per second for 30 minutes

Each admission is a one-unit tick. Each run also checks that every acknowledged admission is billed exactly once, and that no admission inside its caps is refused. A run that misses any service level or either check fails. LOAD_SECONDS shortens or lengthens each leg.

These are the limits the tests measure, not ceilings that the core enforces. Your own limits depend on the host, the disk and the funding mode. Stablecoin and credit card budgets add the chain's and the card processor's latency to settlement.

Watching them

Scrape GET /metrics with CORE_METRICS_TOKEN (the owner token does not scrape; without the token the route is not served), name the scrape job dynamo-core, and load the alert rules in deploy/observability/alerts.yml. make test-observability fires every rule on test series, shows each one quiet on healthy series, and checks that a live scrape exports every metric the rules read:

AlertFires when
DynamoAvailabilityLowmore than one request in a thousand, on any route, answers 5xx over 5-minute windows, for 10 minutes
DynamoAdmissionLatencyHighp95 admission latency, measured in the core, is above 250 ms for 10 minutes
DynamoSettlementLagHighthe settlement lag is above 15 minutes for 10 minutes
DynamoSchedulerLeaseLosta running scheduler has not held its lease for 2 minutes
DynamoSchedulerNotRunningcore hosts are open but no scheduler holds a lease, for 5 minutes
DynamoDutyBackloga scheduler duty has not started for more than three of its intervals plus 60 seconds, for 10 minutes
DynamoDutyFailingone scheduler duty failed more than 3 times in 15 minutes
DynamoRailClaimFaileda payee claim on an adopted rail failed or was lost in the last 30 minutes
DynamoCoreDownthe core does not answer its scrape for 2 minutes

The availability alert counts every request, while the service level counts admissions, so the alert can fire on errors outside admission. On a multi-tenant deployment the scheduler lease metric reads 0 even while tenant duties run (see known limits); watch the duty-run counters there.