Run your instance
This page is for the person who runs a dynamo-core instance: the scheduler, keys at rest, backups, the control console and alerts. It describes the core at owner-api 1.11.0. The boot below runs in Control Mode (no money moves); to add the stablecoin mode on a local chain, see A stablecoin core on a local chain. For the hosted preview, see Preview.
There are two ways to start a core. node deploy/image/entry.mjs (in the image, node /app/server.cjs) is the hosted entry: it runs the production checks below, serves /healthz, rate-limits each client, writes a payload-free audit log, and has the rekey command and the multi-tenant mode. The plain core-host entry, node packages/core-host/dist/main.js after a build, is a development host without those checks. Use the hosted entry for anything another person depends on. The service levels and the alerts are in service levels and limits, a status page draft is in status page, and personal-data export, deletion and the admin audit trail are in privacy.
Boot a local instance
DYNAMO_REPO is the path of your clone of the repository; set it once with export DYNAMO_REPO=/path/to/your/clone. The command below stays in the foreground; a successful boot ends with a line naming owner-api/1.11.0 + plugin wire on http://127.0.0.1:8790.
cd "$DYNAMO_REPO"
pwd
pnpm install --frozen-lockfile
pnpm --filter @dynamo/core-host... -r build
export DYNAMO_OWNER_TOKEN="dp-local-owner-$(openssl rand -hex 16)"
[ -s .dynamo-local.key ] || (umask 077 && openssl rand -hex 32 > .dynamo-local.key)
export CORE_KEY_ENCRYPTION_KEY="$(cat .dynamo-local.key)"
OWNER_API_TOKEN="$DYNAMO_OWNER_TOKEN" PLUGIN_WIRE_TOKEN="dp-local-wire-$(openssl rand -hex 16)" \
OWNER_API_PORT=8790 CORE_DB="$PWD/.dynamo-local/core" \
node deploy/image/entry.mjs
The first boot writes a random key-encryption key to .dynamo-local.key (gitignored); later boots, and the plugin READMEs' quickstarts, reuse it for the same .dynamo-local/ database. A boot with another key refuses with KeyEncryptionError: key check value mismatch (wrong key-encryption key), followed by the database path and the fix: use the key the database was made with, or start over with rm -rf .dynamo-local .dynamo-local.key (that deletes the local budgets). On a real deployment, keep CORE_KEY_ENCRYPTION_KEY somewhere other than the data volume. Without it the instance cannot read its own keys, and its backups cannot be restored.
A stablecoin core on a local chain
make dev-chain starts a local Anvil with a one-second block time, deploys contract version 2 with a test stable, mints it to the budget owner (Anvil dev account 1) and the adopt user (account 4), and prints the environment block to add to the boot command. It stays in the foreground; Ctrl-C stops Anvil. It needs Foundry (anvil, forge) and a built engine.
cd "$DYNAMO_REPO"
pwd
pnpm --filter @dynamo/engine... -r build
make dev-chain
In a second terminal, boot the core with the printed block added (CORE_ALLOWED_MODES=none,usdc, CORE_ESCROW_SETTLEMENT=payee-kit, CORE_ESCROW_RPC_URL, CORE_ESCROW_OWNER_KEY and the CORE_ESCROW_V2_* settings: the stablecoin rail on contract version 2 alone). Every key it prints is a public Anvil dev key; never use them anywhere else. Use a fresh CORE_DB each time you restart make dev-chain, because the new chain does not have the old budgets.
Anvil must run with a block time. On Anvil's default (a block only when a transaction arrives) the latest block can be older than a voucher's signing time, and a settle reverts (SignedInFuture, which tools without the vault ABI show as custom error 0xd663ef37). make dev-chain passes --block-time 1; on your own Anvil, pass it too, or mine a block (cast rpc evm_mine) before settling.
Create a budget
With the instance above running, in a second terminal with the same DYNAMO_OWNER_TOKEN exported. Amounts are micro-USD as decimal strings; 5000000 is 5 USD, in Control Mode (no money moves).
cd "$DYNAMO_REPO"
pwd
curl -s -X POST http://127.0.0.1:8790/owner/v1/budgets \
-H "authorization: Bearer $DYNAMO_OWNER_TOKEN" -H "content-type: application/json" \
-d '{"funding": "none", "capTotal": "5000000", "mandate": {"categories": ["research"], "maxDepth": 3}}'
The same from the Python SDK, after the install step in packages/sdk-python/README.md:
cd "$DYNAMO_REPO"
pwd
DYNAMO_CORE_URL=http://127.0.0.1:8790 .venv-dynamo/bin/python - <<'PY'
import asyncio, os
from dynamoprotocol import Dynamo, usd
async def main():
async with await Dynamo.open(os.environ["DYNAMO_CORE_URL"], os.environ["DYNAMO_OWNER_TOKEN"]) as dyn:
budget = await dyn.open_budget("none", usd("5"), {"categories": ["research"], "maxDepth": 3})
print(budget.budget_id)
asyncio.run(main())
PY
Streams, delegation, freeze, approvals and revocation are in the Python SDK README, packages/sdk-python/README.md in the repository.
Keys at rest
Session keys and the engine signer key are encrypted with AES-256-GCM under CORE_KEY_ENCRYPTION_KEY: 32 bytes, as hex or base64.
- The hosted entry refuses to boot without it. core-host on a loopback address
accepts CORE_ALLOW_PLAINTEXT_KEYS=1 for development only; the hosted entry refuses that variable outright.
- On the first boot with a key, every plaintext key row is encrypted in one
transaction, and freed pages are zeroed so no plaintext remains in the database file. Running it again changes nothing.
- A wrong key refuses to open the database (the stored check value does not
match). A database that holds encrypted keys refuses to open without a key.
- The boot refusal names the database path and the fix, never the key. The
backup, restore and rekey commands never print a key.
- To rotate the key, stop the instance and run
node deploy/image/entry.mjs rekey <new-key-file> [--backup <file>]... (hosted entry only), then boot with the new key. rekey refuses while the instance's scheduler holds a live lease. It cannot see a running multi-tenant deployment, so stop that one yourself. If it stops part-way, boot refuses until you run the same command again.
Before upgrading an existing instance to 1.4.0, set CORE_KEY_ENCRYPTION_KEY on it.
Boot checks, health and metrics
The hosted entry refuses to boot, naming the reason, when:
- a money mode's settings are present but the mode is not in
CORE_ALLOWED_MODES (default none, Control Mode only);
PLUGIN_WIRE_TOKENis empty (single tenant);CORE_KEY_ENCRYPTION_KEYis empty, orCORE_ALLOW_PLAINTEXT_KEYSis set;CORE_SCHEDULER=off;CORE_METRICSis anything but unset oron, orCORE_METRICS_TOKENis
shorter than 32 characters or equal to the owner token;
CORE_LOGis anything butjson(the default) oroff.
GET /healthz answers without a credential with the service name and the owner-api version only. Only the hosted entry and the multi-tenant router serve it. GET /metrics serves Prometheus metrics to CORE_METRICS_TOKEN only, never to the owner token, and only when it is set, on a single core and on a multi-tenant deployment alike (metrics are recorded either way). Its series are deployment totals with no tenant, budget or stream label. The hosted entry limits each client to CORE_RATE_LIMIT_RPS requests per second (default 20) with a burst of CORE_RATE_LIMIT_BURST (default 40), and answers 429 past it.
The scheduler
The scheduler runs the duties the money layer depends on, so none of them waits for a client call. The hosted entry always runs it: it refuses to boot with CORE_SCHEDULER=off. The plain core-host entry accepts CORE_SCHEDULER=off and only prints a warning that nothing signs, settles or expires on its own; never run a deployment that way. Each duty has its own interval, in seconds:
| Duty | What it does | Registered | Variable | Default |
|---|---|---|---|---|
engine.<mode> | due signing, policy checkpoints, checkpoint-band checkpoints, finality polling | one per funding mode | CORE_SCHEDULER_TICK_SECONDS | 15 |
verified.<mode> | verified-mode settlements and finality | with CORE_VERIFIED=1 | CORE_SCHEDULER_VERIFIED_SECONDS | 30 |
control.<mode> | approval timeouts, freeze expiries, backend revoke retries | one per funding mode | CORE_SCHEDULER_CONTROL_SECONDS | 5 |
card | credit card rail: renew holds near expiry, settle each epoch | with the credit card rail | CORE_SCHEDULER_CARD_SECONDS | 300 |
engine.reservations | release reservations held longer than the reservation TTL | always | CORE_SCHEDULER_RESERVATIONS_SECONDS | 60 |
escrow.release-held.<mode> | release matured contract version 2 holdback to payees through the relayer | with contract version 2 | CORE_SCHEDULER_RELEASE_HELD_SECONDS | 60 |
rails.payee-claims | claim payees' vouchers on an adopted rail before its deadline | only where an adopted rail is wired into the core | CORE_SCHEDULER_RAILS_SECONDS | 30 |
rails.payer-exits | refuse a payer exit while a voucher is unclaimed | only where an adopted rail is wired into the core | CORE_SCHEDULER_RAILS_SECONDS | 30 |
alerts | alert evaluation | always | CORE_SCHEDULER_ALERTS_SECONDS | 30 |
mail | send queued alert email, retry failed sends | always | CORE_SCHEDULER_MAIL_SECONDS | 30 |
delivery | webhook delivery | with EVENTS_DB | EVENTS_DELIVER_INTERVAL_SECONDS | 5 |
maintenance | request-log retention, session pruning, escrow reconciliation | always | CORE_SCHEDULER_MAINTENANCE_SECONDS | 3600 |
These duties run on the alerts interval (CORE_SCHEDULER_ALERTS_SECONDS): main-switch (finish an interrupted pause or resume, then the automatic check) and tasks, always; payees.connect and card.payee-held with the credit card rail; kya.refresh when KYA is on; ramps.onramps when ramps are on; and vault.guard with a stablecoin mode. The hosted entry prints every registered duty with its interval on the scheduler on: boot line. No adopted rail is wired in by either entry from the environment.
CORE_CARD_RENEW_WITHIN_SECONDS (default 86400) sets how close to expiry a hold is renewed, and CORE_CARD_EPOCH_SECONDS (default 86400) sets the settlement epoch.
CORE_RESERVATION_TTL_SECONDS (default 3600, minimum 60) is how long a reservation may stay held when it has no per-hold expiry. The reservations duty releases older holds and sends a reservation.expired webhook event for each. A confirm that arrives after the release is billed if the stream's cap still allows it. Otherwise it is refused with reservation_expired, nothing is billed, and an alert fires.
CORE_MAX_HOLD_SECONDS (default 7 days) is the ceiling on an optional holdUntilSeconds passed at reserve (owner-api 1.11.0). A hold with holdUntil stays held past the reservation TTL until that instant, a verified settlement, or stream close; the duty skips it while holdUntil is still in the future. Boot refuses when CORE_MAX_HOLD_SECONDS is below the reservation TTL.
Alert email goes through a queue in the host database. Each alert sends at most one email. A failed send is retried with exponential backoff, from 3 minutes, up to 8 attempts over about six hours, then marked failed. A restart never drops a queued email. An email is recorded as sent right after the mail server accepts it, so only a crash in the moment between the two can send it a second time.
- A duty that is still running skips its next tick, and the skip is logged.
- A stream whose duty fails backs off exponentially, from 5 seconds up to one
hour, and raises an alert.
- One scheduler per database: a second process on the same database cannot
take the lease and refuses to start (scheduler_conflict), so one instance is one process over one data directory.
- With
CORE_TENANCY=multieach tenant's store keeps its own lease, and one
fair loop visits CORE_TENANT_SCHEDULER_VISITS tenant stores (default 16) every CORE_TENANT_SCHEDULER_POLL_MS (default 1000).
- On SIGTERM the running duty finishes, the lease is released, then the
process exits.
GET /owner/v1/schedulershows each duty's last run, duration and error;
POST /owner/v1/scheduler/run runs the duties once, now.
Holdback
CORE_HOLDBACK_DEFAULT_SECONDS (default 300, allowed 0 to 86400) is the holdback window for verified-mode streams that do not set their own. Inside the window a refund executes against held funds. After it the units are final: the scheduler marks them final and fires settlement.final, and a later refund is refused by the protocol.
Backup and restore
Run both commands with the instance's own CORE_DB, CORE_KEY_ENCRYPTION_KEY and, when the instance uses one, EVENTS_DB. They need no owner token or wire token.
cd "$DYNAMO_REPO"
pwd
mkdir -p "$PWD/.dynamo-backups" "$PWD/.dynamo-restored"
CORE_DB="$PWD/.dynamo-local/core" \
node deploy/image/entry.mjs backup "$PWD/.dynamo-backups/core-backup.bin"
# Stop the original instance, then:
CORE_DB="$PWD/.dynamo-restored/core" \
node deploy/image/entry.mjs restore "$PWD/.dynamo-backups/core-backup.bin" --original-stopped
In the image the commands are node /app/server.cjs backup <file> and node /app/server.cjs restore <file> [--original-stopped]. The plain core-host entry has the same two commands for a single-tenant instance only.
- A backup is an SQLite online backup of every database of the instance
(each funding mode, the host store, events), taken while it runs. It carries a SHA-256 digest per file and is encrypted with the key-encryption key.
- A restore refuses a target that already holds data, and refuses a bundle
that is altered, truncated or encrypted under another key.
- Stop the original instance before you boot a restored copy: both hold the
same budgets and keys. A restore drops the scheduler lease it copied. The restored copy's first boot refuses until you confirm the original is stopped, either with restore <file> --original-stopped or by setting CORE_RESTORE_ORIGINAL_STOPPED=1 on that first boot. The confirmation is recorded in the host store, and later boots need nothing.
- A restore that is killed part-way leaves a marker. The instance refuses to
boot while the marker exists, and the next restore clears exactly the files the marker names before it starts.
A multi-tenant deployment
Run the same commands with CORE_TENANCY=multi (and CORE_TENANCY_ROOT when you set one). The bundle holds the tenant registry, every tenant's own stores and the events database. Each tenant's data key stays wrapped under CORE_KEY_ENCRYPTION_KEY inside the bundle; the commands never unwrap one.
- Backup refuses a key that does not open the tenant registry.
- Restore needs the same
CORE_KEY_ENCRYPTION_KEY, or a bundle you re-wrapped
with rekey <new-key-file> --backup <file> when you rotated the key. A wrong key refuses before anything is written.
- Restore drops every tenant store's copied scheduler lease. The restored
deployment's first boot refuses until you confirm the original is stopped (--original-stopped, or CORE_RESTORE_ORIGINAL_STOPPED=1 on that boot); the confirmation covers every tenant, including those first opened later.
- A single-tenant bundle does not restore into a multi-tenant deployment, nor
the other way round.
The control console
The console is served at /console when the instance finds the console bundle: booted from the repository, or with CORE_CONSOLE_DIR pointing at a copy of it. The container image does not carry the bundle, and its boot prints console: not served. Set CORE_CONSOLE=off to turn the console off. Sign in with the owner token.
- The session cookie is HttpOnly and SameSite=Strict, and Secure when
CORE_TLS=1. Sessions last CORE_CONSOLE_SESSION_SECONDS (default 8 hours). Every control action needs the session's CSRF token.
- The console's content security policy allows scripts from the instance
itself only, and no request or response bodies are shown.
- Views: dashboard (spend against caps per budget), the budget page with its
network map (state, frozen, stopped and pending approvals per node), the approval inbox, the request log, the revenue switchboard, scheduler status, reconciliation and alert settings.
- Controls: revoke a node or branch, freeze and thaw, emergency stop per budget
(you type the budget id to confirm), tighten a cap or mandate (widening is refused in the form and by the core), approve and reject. Each one writes a signed authority record.
- The budget page's "Download evidence" button saves the budget's evidence
file (GET /owner/v1/budgets/{id}/evidence), which anyone can check offline with dynamo-verify.
Alerts
Alerts fire on cap thresholds (50, 80 and 100 percent of a budget's cap), spend velocity, a new payee, a mandate refusal, a failed settlement, an approval request, and a late reservation confirm refused after the TTL. Each budget's settings are at GET and PUT /owner/v1/budgets/{id}/alerts and on the budget page.
Alerts go out as signed webhook events (the events system) and, when SMTP is set, as email:
| Variable | Meaning |
|---|---|
CORE_SMTP_HOST | SMTP server; email is off when unset |
CORE_SMTP_PORT | default 25, or 465 with CORE_SMTP_SECURE=tls |
CORE_SMTP_SECURE | none (default) or tls |
CORE_SMTP_USER, CORE_SMTP_PASS | optional login |
CORE_SMTP_FROM | sender address (required with a host) |
CORE_ALERT_EMAIL_TO | comma-separated recipients (required with a host) |
POST /owner/v1/webhooks/test {"type": "cap.threshold"} sends a signed test event, and so does dynamoprotocol trigger cap.threshold. Alert email names budget and stream ids, amounts and reason codes only.
Request log
GET /owner/v1/logs?since=&after=&limit= returns the payload-free request log: time, method, route, status, duration, ids and reason codes. It never holds bodies, credentials, tokens or raw IP addresses. Entries are kept for CORE_REQUEST_LOG_RETENTION_SECONDS (default 7 days). dynamoprotocol logs tail --follow follows it from a terminal.
Refusal retention
A refused delegation (a mandate, depth or category refusal) is recorded as an unsigned diagnostic row and listed under refusals in a budget's tree and network views. Rows older than CORE_REFUSAL_RETENTION_SECONDS (default 30 days, at least 7 days; a smaller value refuses to boot) are deleted by the hourly maintenance duty, with their alert keys. A client pinned at Dynamo-Version 1.11.0 sees the window as refusalsWindowSeconds. Signed authority records and evidence bundles never include these rows, so nothing signed is lost.
Admin audit and privacy
GET /owner/v1/admin-audit (owner-api 1.9.0) is the per-actor trail of console and admin actions. Retention is CORE_ADMIN_AUDIT_RETENTION_SECONDS (default 90 days). Personal-data export and deletion, and what signed records are kept, are in privacy.
Service-node protections
Each payee (service node) is protected from a payer's unsettled spend by the engine defaults: an unsettled-exposure ceiling of 1,000 USD, and headroom bands on the budget. With more than half of the budget left, work runs in full; between a quarter and a half, each request is size-limited; between a tenth and a quarter, a stream must be checkpointed before it takes more work; below a tenth, new work is refused. On every engine pass the scheduler checkpoints each stream in the checkpoint band that has unsettled work of its own. The engine pass runs every CORE_SCHEDULER_TICK_SECONDS (default 15 seconds), so in that band a request refused with checkpoint_pending is admitted again within one engine pass, with no checkpoint call from the client. The refusal carries a Retry-After header; the SDKs and the OpenAI Agents plugin wait and retry it for up to 30 seconds by default.
CORE_NODE_PROTECTIONS sets these per node: a JSON object of service-node address to a full protection set (money as decimal strings, in micro-USD):
{
"0x6a4d0000000000000000000000000000000006a4": {
"maxUnsettledExposure": "5000000",
"maxConcurrentUnpaidWork": 32,
"finalityRequiredAboveAmount": "100000000",
"degradation": { "fullAbovePercent": 0, "throttledAbovePercent": 0, "checkpointRequiredAbovePercent": 0 },
"throttleMaxUnitsPerRequest": "1000000"
}
}
Turn the bands off only for nodes you run yourself, where payer and payee are the same operator and every call is reserved before it runs (the guard does this for its own node). Unknown keys, missing fields or bad thresholds refuse boot. Nodes not listed keep the defaults.
Other settings
CORE_DOCS_BASE_URL: where thedoc_urlof each error points (default
the in-repo error pages).
CORE_ESCROW_RPC_URLS: an ordered, comma-separated list of RPC endpoints
for the escrow backend; later entries are used when earlier ones fail.
CORE_ESCROW_POLL_MS: for a local development chain only (chain id 31337
or 1337), how often the escrow backends poll for receipts, 10 to 4000 ms (default: the client's 4 s). With it set, a backend connected to any other chain refuses to start.
OTEL_EXPORTER_OTLP_ENDPOINT: turns on OpenTelemetry spend spans and
metrics, with payload-free attributes only.
GET /owner/v1/reconciliation: engine-settled against vault-settled per
escrow stream. A mismatch raises an alert and is never corrected automatically.
Try the whole flow
One command runs the operator flow end to end on a fresh local instance (Control Mode, no money, no key, a local mail sink, nothing sent outside the machine): a three-level agent network on the OpenAI Agents plugin with a scripted model, the console, the alert email and signed webhook at 80 percent, a freeze, an approval, an emergency stop, a kill -9 of the core with a reboot, and scheduler-driven finality.
cd "$DYNAMO_REPO"
pwd
make demo-human-control
It needs Node 22.12 or later, pnpm and Python 3.10 or newer; the first run provisions the plugin's virtual environment. Its script, scripts/human-control.mjs, shows every environment variable and console call it uses.
See known limits before you rely on any of this.