Transport-policy layer for rate-limit handling#

  • Status: Accepted

  • Date: 2026-06-02

  • Deciders: saia-python maintainers

Context and Problem Statement#

SAIA enforces layered rate limits (per minute / hour / day / month). A 429 currently aborts the call: ADR-0002 deliberately made the library passive — it surfaces RateLimitInfo and lets the caller decide. So every batch or unattended consumer hand-rolls its own retry/backoff, duplicated and usually wrong (fixed sleeps, no jitter, no awareness of which window was hit, and prone to blocking for ~an hour on the daily window). The driving case is the AvorMedical guideline-ingestion pipeline, which reliably trips the 30/min limit.

Two dispatch-boundary policies already existed in ad-hoc form: the v0.5.0 control-plane timeout wrapper (ArcanaService._request), and the raise_for_status mapping that turns a 429 into RateLimitError already carrying reset_seconds. Should the library act on a 429; where should that logic live; and what should the default be?

Decision Drivers#

  • One owner for retry, not per-consumer boilerplate that drifts.

  • The common batch path (chat / convert) is idempotent → safe to auto-retry.

  • Never block for a long window inside a library call.

  • Preserve the RateLimitError contract and the ADR-0002 metadata.

  • Thread-safe under ADR-0004 workers; an injectable sleep so tests never block.

  • Do not add a third parallel dispatch mechanism.

Considered Options#

  • (a) Stay passive (ADR-0002 status quo) — the caller retries.

  • (b) Opt-in retry (default off).

  • (c) Default-on retry at a single shared dispatch seam — scoped to idempotent calls, opt-out.

  • (d) Per-consumer retry — the avor-backend stop-gap.

  • (e) Session-mounted HTTPAdapter — retry/timeout on the shared session.

Decision Outcome#

Chosen: (c). A thin execute() seam in saia_python/_http.py applies a RetryPolicy at the single point every service dispatches through. Retry is on by default and opt-out, scoped to idempotent calls.

  • The seam sits before raise_for_status and returns the raw response, so when retry is disabled, the budget is spent, or the window must not be waited on, the existing RateLimitError flow fires unchanged — no new exception type, and streaming and non-streaming behave identically.

  • Honest wait policy for the single reset_seconds across four windows: honor a reset within max_waiting_time (default 60 s, settable); fail fast (raise, no sleep) on a longer or zeroed window; bounded blind fallback (31 s × 2) when no reset header is present. The library never blocks ~an hour.

  • Idempotency is declared at the call site; mutations pass idempotent=False and are never auto-retried (overridable per call). Streaming retries only the opening 429, never mid-stream.

  • The control plane shares the same seam (_request now delegates to execute): the timeout default is preserved, idempotent ARCANA reads inherit the policy, and heartbeat opts out so a liveness probe never blocks.

  • Default-on is accepted now while the known user base is tiny (pre-1.0) — the cheapest moment to move a default before adoption hardens expectations.

Consequences#

  • Good — batch jobs survive transient minute-window limits unattended with zero consumer code; a single dispatch seam now carries both timeout and retry; the ADR-0002 metadata is still surfaced on every response.

  • Trade-off — a 429 on an idempotent call now blocks (bounded by max_waiting_time and the retry caps) instead of raising immediately. This is a real behavior change, accepted for the small user base; opt out with SAIAClient(retry=False) or per call.

  • Trade-off — only the minute window’s reset is timable from a single reset_seconds; longer windows fail fast rather than waiting precisely.

Confirmation#

tests/test_transport_policy.py covers the planner, execute, the streaming/non-streaming integration, and resolve_retry; tests/test_arcana.py::TestControlPlaneRetry covers the mutation / heartbeat / idempotent wiring. All waits route through an injected sleep, so the suite never blocks. The unchanged TestDefaultTimeout confirms the control-plane timeout behavior is preserved by the convergence.

More Information#

Amends ADR-0002 — the passive default becomes active (opt-out); the metadata decision itself stands. Extends ADR-0004 — retry runs inside the wait=False worker and shared state is thread-safe. Folds in the v0.5.0 control-plane timeout wrapper as the control-plane instance of the same seam. The full design and phased rollout live in docs/proposals/rate-limit-handling.md.