Bursts become provider throttles.
Every workload races for the same finite pool. Interactive requests fail beside work that could have waited.
Open-source inference QoS
Finite AI capacity, scheduled fairly.
Protect interactive traffic, queue flexible work, and measure whether another capacity block is actually necessary.
0required external services
5built-in service classes
4provider families
Apache-2.0open-source core
The operating problem
You paid for a fixed inference envelope. Demand does not arrive smoothly.
Request counters cannot distinguish a small chat turn from a 100,000-token workflow. InferQoS estimates normalized work before dispatch and admits it against the capacity that actually exists.
Urgency, fairness, deadlines, and work size shape admission.
Unused shares remain borrowable. New high-value traffic reclaims newly available capacity without pretending running generations can be preempted.
Replay the metadata you already have.
Compare baseline behavior with configured QoS and receive an honest answer when sustained saturation means scheduling cannot help.
A control plane for admission, not another AI gateway.
Keep APIM, Kong, Envoy, or your existing gateway. Change a base URL or route, add authenticated QoS metadata, and preserve OpenAI-compatible streaming end to end.
Integration patternsApplicationsuntrusted clients
Existing gatewayoptional
InferQoS
- Identity and policy
- Work estimate
- Fair admission
- Capacity feedback
AzureAWSGCPCompatible API
Small runtime. Serious control.
Rust-first, work-conserving, cancellation-aware, and production-capable with no database or broker. Add Valkey only when replicas share one finite pool.
Hierarchical fairness
Weighted service classes, tenants, and applications share capacity by normalized work. Deadline pressure and queue aging prevent rigid priority from becoming starvation.
Adaptive capacity model
Predicted and actual token use, cached input, latency, 429s, and Retry-After tune a conservative safety factor.
Shadow first
Calculate queue, rejection, and SLO outcomes without delaying production traffic. Enforcement is one configuration change.
Adoption pathStreaming stays streaming
SSE responses pass through without full buffering. Client cancellation releases claims and propagates upstream when safe.
Correct HA
Atomic Valkey reservations and expiring leases coordinate replicas. Loss of coordination fails closed by default.
Put FIFO, strict priority, and InferQoS under the same workload.
Change capacity, burst intensity, and workload mix. The deterministic browser simulator shows queue latency, throttling, SLO attainment, fairness, and utilization side by side.
Open the simulatorOne normalized model, four production adapter families.
Provider-specific authentication, throttling signals, token accounting, and capacity semantics stay inside adapters. The scheduler never assumes every token costs the same.
Azure OpenAI
Provisioned deployments, managed identity, Retry-After variants, usage reconciliation, and background Azure Monitor calibration.
AWS Bedrock
Provisioned throughput patterns with ambient AWS credential resolution and native SigV4 signing.
Google Vertex AI
Provisioned throughput with Application Default Credentials and workload identity.
OpenAI-compatible
Any finite compatible endpoint with configurable work coefficients, custom headers, TLS, streaming, and 429 learning.
Private by default
Your inference path should not become our analytics feed.
No project analytics. No anonymous analytics. No prompt or completion logging. No outbound telemetry except the provider and observability endpoints you explicitly configure.
- Identity
- OIDC, mTLS, API-key mapping, trusted proxy boundaries
- Resources
- Bounded queues, body limits, secure local spool, concurrency guards
- Supply chain
- Signed OCI images, SBOM, provenance, checksums, pinned CI actions
- Runtime
- Non-root, rustls, static upstreams, no arbitrary per-request routing
Zero keys. Zero required services.
The local demo starts a deterministic finite provider, InferQoS, an example client, and the operational dashboard.
git clone https://github.com/dlamaro96/inferqos.git
cd inferqos
just demo