Service levels and limits
This page states the service levels a dynamo-core deployment is built and tested to, the limits they were measured at, and the metrics and alerts that watch them. The load test (make test-load) measures every one of them on a local deployment. They are design targets, not commitments: the core does not enforce them, and they are not an offer of service to anyone. What a status page can show from the same metrics is on status page.
Service levels
| Deployable | Availability | Admission latency (p95) | Settlement lag |
|---|---|---|---|
| dynamo-core, one tenant | 99.9 % of admissions answered without a server error | 250 ms or less | 15 minutes or less |
| dynamo-core, many tenants (the router) | 99.9 % of admissions answered without a server error | 250 ms or less | 15 minutes or less |
- Availability counts the admissions (stream opens, child opens, ticks, meters, reservations and their confirmations) that are answered without a 5xx or a network error. A refusal at a cap is an answer, not an outage.
- Admission latency is measured from the request to its answer, for success and refusal alike. The load test measures it at the client; the metric is
dynamo_admission_duration_seconds, measured in the core. - Settlement lag is the age of the oldest signed voucher that no checkpoint covers yet. The metric is
dynamo_settlement_lag_seconds, by funding mode. On a stablecoin budget it includes the time the chain takes to mine the checkpoint.
Tested limits
make test-load runs two legs, one tenant and many tenants, against a deployment booted from the repository, in Control Mode:
| Profile | One tenant | Many tenants |
|---|---|---|
| Reduced: part of the full gate | 20 budgets × 5 streams, 100 admissions per second for 20 seconds | 3 tenants × 4 streams, 30 admissions per second for 15 seconds |
Full: the soak, started by hand (LOAD_PROFILE=full) | 200 budgets × 10 streams, 300 admissions per second for 30 minutes | 20 tenants × 10 streams, 100 admissions per second for 30 minutes |
Each admission is a one-unit tick. Each run also checks that every acknowledged admission is billed exactly once, and that no admission inside its caps is refused. A run that misses any service level or either check fails. LOAD_SECONDS shortens or lengthens each leg.
These are the limits the tests measure, not ceilings that the core enforces. Your own limits depend on the host, the disk and the funding mode. Stablecoin and credit card budgets add the chain's and the card processor's latency to settlement.
Watching them
Scrape GET /metrics with CORE_METRICS_TOKEN (the owner token does not scrape; without the token the route is not served), name the scrape job dynamo-core, and load the alert rules in deploy/observability/alerts.yml. make test-observability fires every rule on test series, shows each one quiet on healthy series, and checks that a live scrape exports every metric the rules read:
| Alert | Fires when |
|---|---|
DynamoAvailabilityLow | more than one request in a thousand, on any route, answers 5xx over 5-minute windows, for 10 minutes |
DynamoAdmissionLatencyHigh | p95 admission latency, measured in the core, is above 250 ms for 10 minutes |
DynamoSettlementLagHigh | the settlement lag is above 15 minutes for 10 minutes |
DynamoSchedulerLeaseLost | a running scheduler has not held its lease for 2 minutes |
DynamoSchedulerNotRunning | core hosts are open but no scheduler holds a lease, for 5 minutes |
DynamoDutyBacklog | a scheduler duty has not started for more than three of its intervals plus 60 seconds, for 10 minutes |
DynamoDutyFailing | one scheduler duty failed more than 3 times in 15 minutes |
DynamoRailClaimFailed | a payee claim on an adopted rail failed or was lost in the last 30 minutes |
DynamoCoreDown | the core does not answer its scrape for 2 minutes |
The availability alert counts every request, while the service level counts admissions, so the alert can fire on errors outside admission. On a multi-tenant deployment the scheduler lease metric reads 0 even while tenant duties run (see known limits); watch the duty-run counters there.