SSC-STUDIO / Ai-Model-Gateway

LLM Provider Fallback Gateway

Route OpenAI-compatible LLM traffic around upstream quota, timeout, and outage signals while keeping provider keys, fallback policy, telemetry, config publishing, and rollback inside your own environment.

View on GitHub Read fallback operations guide Run fallback demo

If this solves a real self-hosted provider failover workflow, star the repository after evaluation.

Failure modes

Provider fallback is an operations problem, not just a retry loop.

LLM upstreams fail through quota exhaustion, 429 bursts, slow endpoints, transient 5xx responses, model-level outages, and bad routing changes. AI Model Gateway keeps those decisions observable and reversible in the gateway layer.

Health-aware routing

Use provider health and routing policy to avoid unhealthy upstreams while preserving one local gateway endpoint.

Fallback telemetry

Record route mode and provider behavior so operators can see when traffic used a fallback path.

Cooldown state

Keep degraded providers out of rotation long enough for incidents and quota windows to clear.

Config rollback

Preview, diff, publish, audit, and roll back routing changes when a fallback policy behaves badly.

Executable proof

Verify failover locally before wiring real providers.

The provider fallback demo starts two fake OpenAI-compatible upstreams. The primary returns 429, the gateway serves the request through a fallback provider, rewrites the forwarded model, and records route_mode=model_fallback.

go test ./examples/provider-fallback -run TestProviderFallbackDemo -v
Open the fallback demo
AI Model Gateway monitoring workspace showing provider and traffic telemetry

Runbook shape

A fallback gateway should support the whole incident loop.

  1. Detect. Use provider health, latency, error, cost, and request telemetry to spot degraded upstreams.
  2. Route. Send traffic through fallback policy without changing every OpenAI-compatible client.
  3. Inspect. Review probes, diagnostics, request logs, route mode, and provider behavior in the Admin UI.
  4. Recover. Publish safer config, audit the change, and roll back quickly if the policy is wrong.
Provider fallback and health guide Config publish and rollback 15-minute evaluation Self-hosted LLM gateway page OpenAI-compatible gateway page LLM gateway comparison page

Fit check

Use it when failover policy needs local ownership.

Good fit

  • You need fallback across multiple upstream LLM providers.
  • OpenAI-compatible clients should keep one stable gateway URL.
  • Provider keys, routing policy, telemetry, and audit records must stay local.
  • Operators need probes, diagnostics, config publish, and rollback during incidents.

Less ideal

  • You only need client-side retry code inside one application.
  • You want a hosted model marketplace to own routing and billing.
  • You do not need provider health, request telemetry, or audit logs.
  • You do not want to operate a local gateway runtime.

Review evidence

Check installability, quality, and security before adopting it.

Release archive install

Try the packaged v1.4.4 runtime with checksum verification, local config, runtime directories, and supervised startup commands.

Open release install path

Quality evidence

Review CI gates, local reproduction commands, runtime smoke checks, feature proof points, and current capability boundaries.

Open quality evidence

Security and trust model

Inspect admin auth, same-origin browser writes, provider-key handling, SSRF defenses, telemetry sensitivity, and update trust.

Open security model

Next step

Test the fallback path, then decide whether it earns a star.

Start with the executable demo and operations guide. If the project matches your self-hosted LLM failover needs, starring the repository helps other operators find it.

Star on GitHub Leave feedback