Client compatibility
Use OpenAI-compatible entry points so existing clients can route through one gateway layer.
Why use it
One local control point for OpenAI-compatible model traffic.
AI Model Gateway is built for teams that want to keep provider keys, routing policy, request telemetry, and incident workflows inside their own environment instead of delegating the control plane to a hosted broker.
Provider fallback
Route around upstream quota, timeout, and failure signals without moving keys or policies out of your environment.
Local rate limits
Keep ingress and provider-facing limits visible alongside model routing, telemetry, and operator workflows.
Rollback path
Preview, diff, publish, audit, and roll back config changes instead of editing a live proxy file.
Operational proof
Test provider fallback before trusting it.
The demo starts two fake OpenAI-compatible upstreams. The primary provider returns 429, the gateway serves the request through the fallback provider, rewrites the forwarded model, and records route_mode=model_fallback.
go test ./examples/provider-fallback -run TestProviderFallbackDemo -v
Fit check
Use it when the gateway itself is part of operations.
Good fit
- Internal teams need one OpenAI-compatible gateway endpoint.
- Provider fallback, cooldowns, and diagnostics matter during incidents.
- Config publishing and rollback need auditability.
- Model traffic, latency, cost, and health should be inspectable locally.
Less ideal
- You only need a hosted model marketplace.
- You do not want to operate local runtime infrastructure.
- You only need a client SDK wrapper without admin workflows.
- You do not need provider health, telemetry, or rollback.
Review evidence
Check installability, quality, and security before adopting it.
Release archive install
Try the packaged v1.4.4 runtime with checksum verification, local config, runtime directories, and supervised startup commands.
Open release install pathQuality evidence
Review CI gates, local reproduction commands, runtime smoke checks, feature proof points, and current capability boundaries.
Open quality evidenceSecurity and trust model
Inspect admin auth, same-origin browser writes, provider-key handling, SSRF defenses, telemetry sensitivity, and update trust.
Open security modelNext step
Evaluate the project, then decide whether it earns a star.
Start with the short evaluation path or provider fallback demo. If the project matches your self-hosted LLM gateway needs, starring the repository helps other operators find it.