litellm-dynamo

Not in the preview yet. This feature is built in the repository but is off in the v2.2.6 preview (test mode). Do not treat this page as available on a preview core. See ../preview.md and the release notes.

Meter and enforce allowances on every model call through a LiteLLM proxy you operate - without adding a network hop to the request path. Each caller presents an allowance credential as their API key; the plugin verifies it in-process against your pinned engine signer. No live credential, no completion: HTTP 402.

Payments are none of the plugin's business. The credential might be backed by a no-money Control Mode budget, a stablecoin budget or a credit card budget - the plugin neither knows nor cares (mode-agnostic by construction). Control Mode is Live; a funded budget runs today on a local chain or in Stripe TEST mode; real money is Release-gated.

Install

litellm-dynamo 1.7.0 is not published; install it from a clone. Python 3.10 to 3.13 (3.14 is refused: its signing library publishes no 3.14 wheel); the LiteLLM pin is 1.104.0 (the litellm extra), which has no Intel macOS wheels - on an Intel Mac run the proxy in the LiteLLM image, as the guard does:

cd "$DYNAMO_REPO/packages/plugin-litellm"
pwd
pip install -e ".[litellm]"

Two environment variables are mandatory (plus the wire bearer on hosted instances):

export DYNAMO_ENGINE_URL=https://your-instance-url   # your dynamo-core - ASYNC leg only
export DYNAMO_ENGINE_SIGNER=0xYourEngineSigner       # PINNED signer; verification is offline
export DYNAMO_ENGINE_TOKEN=your-plugin-wire-token    # bearer for /plugin/v1/* on hosted instances

Choose how units are counted - config, not code. By default (DYNAMO_METERING=tokens) your tokens map onto Dynamo's payload-neutral units:

export DYNAMO_UNITS_PER_PROMPT_TOKEN=1
export DYNAMO_UNITS_PER_COMPLETION_TOKEN=2
export DYNAMO_UNITS_PER_REQUEST=0
# Required: units billed for a delivered response whose usage is missing,
# 0/0 or one-sided (the provider reported only one side).
export DYNAMO_RESERVED_UNITS_PER_RESPONSE=8192

Or meter in money: with DYNAMO_METERING=price, DYNAMO_PRICE_TABLE pointing at a price table and DYNAMO_DEFAULT_MAX_TOKENS set (the plugin refuses to start in price mode without either), one unit is one micro-USD, priced per model from rows that each carry a source and an as-of date. A model with no row is refused with 402 before the upstream is called; there is no default price.

Then register the hook in a dynamo_hook.py beside your proxy config:

from litellm_dynamo import DynamoLiteLLMHook, config_from_env

dynamo_hook = DynamoLiteLLMHook(config_from_env())

The request path, precisely

  1. Pre-call (free): credential parsed and verified offline (JWS ES256K

against the pinned signer), expiry and revocation-at-last-sync checked, estimated units pre-checked against cached headroom. Any failure → 402 with an x402-shaped challenge body. The upstream is never touched.

  1. The model call proceeds only for a live, sufficient allowance.
  2. Post-call (async): actual usage maps to units and is metered to the

engine (POST /plugin/v1/meter) off the request path. Metering never blocks a response. A zero-filled usage report (prompt_tokens and completion_tokens both 0) counts as not reported: in price metering the reservation is billed (entry 137(b)(1)). With DYNAMO_STATE_DIR set, every delivered request is journaled to disk before it is metered, and a journal left by a crash is billed exactly once on restart. The engine's answer is classified by its reason code, as every gateway does: a final refusal (revoked, cap, budget_closed, a mandate or policy code, anomaly, sla, or 402/404/410/422) is parked, never sent again, kept in the journal with its units, logged as meter_job_parked, and halts that stream; a pending one (checkpoint_pending, frozen, …) waits for its Retry-After; an unreachable engine or a 429/5xx keeps the job queued.

There is no fail-open configuration. Missing, stale, exhausted, expired, unverifiable, or revoked-at-last-sync → deny. The denial classes, their evaluation order, and the 402 challenge schema are identical across every gateway surface (Kong, APISIX) - enforced by a shared conformance vector set, so a caller cannot distinguish gateways by their denials.

Verify an install

The quickest live check is the guard - it composes exactly this plugin with a LiteLLM proxy and prints a capped base URL in one command: one completion serves (HTTP 200), spending past the cap answers HTTP 402, and the upstream is not touched after the stop. From the repository root, make test-py runs the plugin's own suite and make demo-litellm meters a model call through the 402 flow in all three funding modes (an offline mock upstream without a provider key).