Known limits (owner-api 1.11.0)
Each limit is marked with what it is: ① not built yet, ② a business choice, ③ waiting on counsel. Where a limit concerns a feature's availability, its status is given as Live (Control Mode or a testnet today), Preview (behind a switch that is off by default) or Release-gated (real money, live credit card keys, operator-mode ramps).
Access and control
- ① Without single sign-on, one owner token. On a single-tenant
instance the console and the owner-api accept the instance's owner token. Organisations, teams and roles (viewer, approver, funder, admin) exist, but a member signs in only through single sign-on: OpenID Connect or SAML 2.0, off by default and tested against local mock providers only (Preview). Without it, anyone holding the owner token can do everything, approvals included.
- ② A role only narrows. A member can do nothing the owner token cannot,
and a viewer moves no money. A member outside a role's scope gets one answer, 403 role_forbidden.
- ② A client-keyed hold is approved by its client. A held stablecoin open
on contract v2 that the SDK keyed completes only with that client's session key, which the core never holds. The console and <dynamo-approvals> show it as "approve from the client that opened it" and offer reject only. If the client lost its key, reject the hold and open the stream again.
- ① No push notifications. Alerts arrive by webhook and email only.
- Many tenants on one deployment.
CORE_TENANCY=multiserves many
tenants from one deployment, each with its own stores, engine signer and data key (Live in Control Mode). A tenant's live keys work in Control Mode only; money modes open for test keys only.
- ② Guards on the same instance share one service node. Every
dynamo-guard bills one fixed service-node address.
Freeze, stop and amendment by funding mode
- ① Stablecoin escrow, contract version 1: freeze is engine-level. The
version 1 contract has no on-chain pause, so a freeze or a freeze-mode stop is enforced by the engine, not by the vault. A revoke-mode stop does call the contract.
- ① **Stablecoin escrow, contract version 1: amendments do not lower the
on-chain cap.** The version 1 contract cannot reduce a cap. The engine enforces the amended cap at every admission and allocation.
- ① Stablecoin escrow, contract version 1: delegation depth 8. The
version 1 contract holds escrow budgets to 8 levels; contract version 2 and the other modes allow 16.
- Contract version 2 has a vault-wide pause only. The vault owner can
pause new budgets, new streams and settles of streams not yet cut off for at most 3 days, with a day between pauses. A pause never blocks release, refunds, revocation or the return of unspent value. While a vault is paused, the core opens nothing on it and those settles wait.
- On every mode, freeze, stop and amendment leave signed work settleable: a
payee is paid for work signed before the action.
Keys
- ① The key-encryption key comes from the environment. The engine has a
key-provider interface for a key management service, but no provider ships; both entry points read CORE_KEY_ENCRYPTION_KEY only.
- Own hops in money modes. In the stablecoin and credit card modes each
hop is signed by its delegator, and each payee signs its own offer (CORE_OWN_HOPS defaults to required there). In Control Mode own hops are optional unless CORE_OWN_HOPS=required is set.
- ① The engine signer cannot be rotated in place. A new signer means a new
deployment; gateways and verifiers then pin the new address.
- ② Registering terms offers needs Dynamo-Version 1.9.0 (R-414). Pin 1.9.0
and read the audience from GET /payee/v1/audience. A 1.8.0 pin is refused terms_offer_version; there is no audience discovery or relaxed body at that pin (entry 138(f)).
- ② A registered terms offer needs an EOA serviceNode key (R-505). The core
keeps the payee's registering proof and checks it offline, at open and in evidence 1.5.1, so registration refuses a contract-account (ERC-1271) signature with terms_offer_invalid; such a payee signs its offer itself (dynamo-terms-offer-v2). An offer registered before this change has no kept proof: opening a stream on it is refused, and the payee registers it again.
Settlement and fees
- ② Holdback default 300 seconds (
CORE_HOLDBACK_DEFAULT_SECONDS). A refund
after the window is refused by the protocol.
- ② Fee levels come from the deployment, not from code. The protocol fee
is read from the stablecoin vault and is never set by the core. The platform fee on the credit card rail starts switched on at 50 bps and charges only while the credit card rail is configured; an operator can change it, and each budget keeps the rate it opened with. The usage fee is off.
- ① Checkpoint band waits for one engine pass. With between a tenth and a
quarter of a budget left, a stream on default protections must be checkpointed before each further request. The scheduler checkpoints such streams on every engine pass (CORE_SCHEDULER_TICK_SECONDS, default 15 seconds), so a request sent right after another is refused with checkpoint_pending and admitted again within one pass. The response carries a Retry-After header, and the SDKs and the OpenAI Agents plugin wait for the pass and retry the same request (30 seconds at most by default; see checkpoint_pending). Below a tenth, new work is refused (see service-node protections).
- ② A reservation is held for at most
CORE_RESERVATION_TTL_SECONDS
(default one hour), unless the reserve named a holdUntilSeconds (owner-api 1.11.0, entry 159(a)). With no per-hold expiry the hold is released after the TTL and its room returns to the budget. A confirm that arrives later is billed if the cap still allows it; if other spending has used the room meanwhile, it is refused with reservation_expired, nothing is billed, and an alert fires. That late work is the one case where delivered work can go unpaid, and it is bounded by the reserved units of each confirm that arrives after the TTL. A hold with holdUntil is kept until that instant (bounded by CORE_MAX_HOLD_SECONDS, default 7 days, and the stream's remaining life), a verified settlement, or stream close - so an x402 V2 authorization or batch voucher can still settle after the TTL.
- ② Money modes are off by default. The hosted entry runs Control Mode
unless CORE_ALLOWED_MODES names the stablecoin mode or the credit card rail, and refuses to boot when a money mode's settings are present without it. The plain core-host development entry has no allowlist: it opens a money mode whenever that mode's settings are present.
- ② Real money is release-gated. The credit card rail runs with Stripe
TEST keys only: CORE_CARD_RAIL_MODE=live stops at card_live_release_gated even with every safeguard in place, and a live-shaped Stripe key anywhere in the environment refuses boot (Release-gated). Operator-mode ramps report their live custody as release-gated (Release-gated). Stablecoin: CORE_ESCROW_RAIL_MODE defaults to test; the hosted entry refuses a non-local chain in test mode (escrow_rail_public_chain), and live needs the audit flag then stops at escrow_live_release_gated (Release-gated). A stablecoin mode off a local chain needs a vault whose owner and fee collector are Safe multisigs with a threshold of two or more. When contract v2 is configured, new budgets default to v2 and v1 opens no new budget unless CORE_ESCROW_V1_STATUS=active.
Gateways
- ② LiteLLM streaming in tokens mode. From litellm-dynamo 1.4.0 each
streamed delivery is written to the plugin's journal (fsynced) before the final chunk and [DONE] reach the client. In price mode a bound is journaled first, so a crash or a client disconnect mid-stream bills at most the reservation (the payer's own stated maximum for that call). In tokens mode there is no bound: if the proxy crashes mid-stream, the content already delivered on that one connection is unmetered. A tokens-mode stream that can be counted neither from provider usage nor by LiteLLM's local count is refused.
- Reused LiteLLM call ids (fixed in DN4 fix pass; library mode fixed in
litellm-dynamo 1.4.0). Each delivery is billed under its own minted ref, whatever call id the client sends.
- Agents through LiteLLM's A2A gateway (from litellm-dynamo 1.5.0). Only
message/send is metered, in price mode, for agents with a price-table row, and each call is billed to a child of the caller's stream. message/stream, tasks/resubscribe and unknown methods are refused.
- A meter the engine refuses for good (litellm-dynamo, like every
gateway). The job is parked, not billed and not sent again: it stays in the journal as a park line with its units, meter_job_parked is logged, and the stream is halted until the proxy restarts. Clearing or settling a parked job is the operator's call; litellm-dynamo never drops it.
- x402 exact payments the seller refuses (Kong, APISIX and the reference
seller). The buyer pays before the seller serves (the engine-signed sale receipt is the seller's proof of payment). If the seller then refuses, or its upstream fails, the bill is not reversed: Dynamo doesn't decide refunds, and an unsigned refusal can't undo a payment. The remedy is to re-present the same paid receipt, which the seller serves once for that sale until the receipt expires (300 seconds after it was issued). x402Fetch re-presents it on the next call to the same request. A receipt that expires unserved stays billed: at most one price per sale.
- Headroom between syncs (Kong, APISIX, Envoy and Bifrost gateways). Each
admission reserves the units the decision table needs (the estimate, never below precheck_min_units) until its response settles, and every worker of a Kong or APISIX node shares the snapshot, the reservations and the units delivered since the last sync through the declared shared dict. Per stream, one node overshoots the cap by at most the sum, over requests in flight, of what each bills beyond its reservation: nothing when each bills at most its estimate, as in price and per-request modes. Several nodes in front of one stream each keep their own view until the next sync. On Kong and APISIX an admission waits at most 0.5 seconds for the shared lock, then is refused, named, with nothing reserved. A node without the shared dict declared keeps each worker's own view.
- Two refusals only Kong and APISIX give (their workers share one dict
under a lock; Envoy and Bifrost are one process each and never give them). When the shared lock stays busy, or the full dict cannot record the reservation, the admission is refused with a 402 in the usual challenge shape: headroom unknown: admission lock unavailable (<error>) (fail closed) or headroom unknown: in-flight units not recorded (<error>) (fail closed). Nothing is reserved. Both are transient and carry Retry-After: 1; x402Fetch waits that second and pays once more. They share the headroom unknown class of every gateway, but no shared conformance vector covers their detail text.
Backup and restore
- A restored copy beside its original (fixed in DN4 fix pass). A restore
drops the scheduler lease it copied, and the restored copy's first boot refuses until you confirm the original is stopped (restore <file> --original-stopped, or CORE_RESTORE_ORIGINAL_STOPPED=1 on that boot). The confirmation is recorded. The core cannot see the original itself, so the confirmation is yours to get right: stop the original first.
- A multi-tenant backup is consistent per database. Each tenant store is
copied with an online backup while the deployment runs, one after another, so two tenants' copies can be a moment apart. Each tenant's own billing is consistent within its store.
rekeycannot see a running multi-tenant deployment. It refuses while a
single-tenant instance's scheduler holds its lease, but a multi-tenant deployment keeps its leases in each tenant's store. Stop the deployment before you rotate its key.
Monitoring
- ① Scheduler lease metrics on a multi-tenant deployment. Tenant
schedulers are driven by one shared loop rather than started one by one, and the lease and duty-backlog metrics are reported only for a started scheduler. On a multi-tenant deployment the lease metric therefore reads 0 and the DynamoSchedulerNotRunning alert can fire while duties run. Watch the duty-run counters there instead (see service levels).
Data
- ③ Retention. The request log keeps 7 days by default, and alert email goes
to the addresses the operator sets. The retention period and the email content are pending counsel review.
Deferred from the v2.2.0 Fable check
Entries 177(e), 178(e) and 179. Preview users stay on test mode; each row names the effect and the mitigation.
| ID | Effect on a preview user | Mitigation |
|---|---|---|
| V220-E1-02 | Unpaid V1 challenges grow seller-v1-refs.json without bound; a flood can stall the seller process | Cap public exposure of V1 challenge routes; restart and prune the state dir if the API stalls; prefer V2 seller paths when possible |
| V220-E1-03 | Facilitator settles after the seller's 30 s timeout: wallet paid, nothing served, stream not billed, seller row stays settling | Configure settlementResolver / chainSettlementResolver; report any settlement_pending that will not clear with its transaction hash; operator can resolve() by hand |
| V220-E1-04 | Two processes that break one stale file lock can both enter (lost row; rare second serve) | Run one seller process per state directory, or one volume per process; do not share the x402 state dir across containers |
| V220-E1-06 | A kept settlement whose late confirm the engine refuses blocks every later x402 payment on that stream | Call resolvePendingV2 with nowSeconds once the hold TTL has passed; tighten caps only when no pending settlements remain |
| V220-E1-07 | Vault-guard and supersede logic assume RPC lag below CORE_ESCROW_CONFIRMATIONS | Use a single reliable RPC; raise confirmations on flaky providers; monitor vault-guard alerts |
| V220-E1-08 | Non-canonical upto nonce spellings can run work once each; a crash leaving verified answers settlement_pending forever | Buyers send one canonical nonce form; sellers call resolve() on stuck verified rows after a crash |
| V220-E2-01 | Nested n8n npm ci can run unlisted install scripts in some gate targets | n8n is off in the preview; do not enable it |
| V220-E2-02 | A seller payment left verified by a crash answers settlement_pending forever | Same as E1-08: resolve() after restart; report stuck keys |
| V220-F-01 crash window | A crash between the serving: claim and the served write can leave a marker that blocks a second serve while work may still be running | Restart the seller; delete only the stuck serving: marker under operator guidance; report the payment key |
| V220-F-02 | A crash between the resolved: claim and the row write can leave a marker; markers are not deleted with their row | Restart and report; operator cleans orphan markers in the state dir |
| V220-F2-01 | Listed deferred by entry 179; the Fable F2 report (V220-F2-FIX.md) was not present in the repository at documentation time | Treat unexpected double-serve or unpaid-serve behaviour as a reportable incident; keep test funds only |