Deployments & Routing

            Provider
               ↓
     Global LLM Collection
               │ (Workspace Access)
               ▼
      Workspace LLM Access
        ├──────────────┐
        ▼              ▼
  Direct Model    Deployment   ← You are here
        \              /
         \            /
          ▼          ▼
         Virtual API Key
               ↓
        Gateway Request

A Deployment is the final, client-facing tier of Infralo's gateway. It serves as a stable, virtual API endpoint that client applications call.

Instead of hardcoding a specific provider model (like gpt-4o) or managing raw API credentials inside your code, your application calls your virtual Infralo deployment (e.g., production-chat-gateway). Infralo then resolves this request and routes it to one or more of your workspace's whitelisted models based on your configured load-balancing, failover, and caching policies.

Gateway Access Paths

When sending requests to Infralo's OpenAI-compatible gateway, you can target models in two ways depending on your needs:

1. Direct Access (via Model Alias)

Use the whitelisted model's registered Alias Name as the model parameter in your API request.

• Behavior: Requests bypass deployment-level load-balancing rules, circuit breakers, and runtime modules. They are routed directly to the single specified LLM.
• Use case: One-off scripts, model-specific testing, or tasks requiring a dedicated target.
• Example request:

  response = client.chat.completions.create(
      model="gpt-4o-prod",  # The Model Alias Name
      messages=[{"role": "user", "content": "Direct connection test"}]
  )

2. Virtual Access (via Deployment Name)

Use the virtual Deployment Name as the model parameter.

• Behavior: The request passes through Infralo's virtualization layer, where load balancing, circuit breaking, failover rules, caching, and Runtime Modules (PRE and POST stages) are automatically applied.
• Use case: Production environments requiring high availability, cost management, data compliance, or prompt styling.
• Example request:

  response = client.chat.completions.create(
      model="production-chat-gateway",  # The Virtual Deployment Name
      messages=[{"role": "user", "content": "Production connection"}]
  )

Load Balancer (LB) Presets

Each deployment is configured with a load-balancing mode that dictates how incoming API requests are distributed. Infralo provides three built-in presets and a fully custom option:

Routing Strategies

Infralo supports five core load-balancing and routing strategies under Custom mode (as well as the underlying preset engines):

1. Priority Routing (Deterministic Failover)

Priority routing implements a deterministic, multi-tier active-standby failover architecture:

• 100% Primary Traffic: All traffic is routed to models configured in Priority 1 (Primary) as long as they are healthy and within rate limits.
• Automatic Multi-Tier Cascading: If Priority 1 models enter circuit breaker cooldown or reach RPM/TPM limits, traffic automatically cascades to Priority 2 (Fallback 1), then Priority 3 (Fallback 2), supporting arbitrary $N$ priority levels.
• Immediate Recovery: As soon as a higher priority tier recovers from cooldown, traffic immediately resumes to it without manual intervention.
• Same-Tier Peer Balancing: When multiple models share the same priority tier (e.g., OpenAI GPT-4o and Azure GPT-4o both configured at Priority 1), traffic is dynamically balanced between them using Least Busy (fewest in-flight requests).
• Gapless Normalization: The backend automatically normalizes non-consecutive priorities (e.g., [1, 3] becomes [1, 2]) to eliminate configuration gaps.

2. Weighted Round Robin (WRR - Ratio Traffic Splitting)

Weighted Round Robin distributes requests proportionally based on configured weights (integers from 1 to 100):

• Exact Ratio Guarantees: For example, configuring Model A with weight 80 and Model B with weight 20 guarantees an exact 80% / 20% traffic split.
• Smooth Interleaving: Interleaves requests evenly (e.g., [A, A, B, A]) instead of bursting [A, A, A, B] to prevent sudden load spikes on single providers.
• Distributed State Synchronization: Rotation state is coordinated across all gateway workers to maintain seamless traffic balancing at scale.
• Non-blocking Bypass: If a model in the rotation sequence is in cooldown or over rate limits, the engine skips it without stalling the rotation or blocking other healthy models.

3. Least Busy (Queue Depth)

Routes each incoming request to the healthy target model with the fewest active in-flight requests:

• Tracks concurrent in-flight requests across all gateway instances in real time.
• Uniformly balances burst traffic across peer models when active request counts are tied.
• Best suited for workloads with high concurrency and variable token generation times.

4. Latency Based (EMA Performance Routing)

Routes requests to the model with the lowest rolling response time:

• Computes a rolling Exponential Moving Average (EMA) latency using your configured smoothing coefficient ($\alpha$):

  EMA_new = (alpha * Latency_request) + ((1 - alpha) * EMA_old)

• Reacts rapidly to downstream provider degradation while smoothing out transient outlier spikes.
• Tie-breaks between equal-latency models using active request counts.

5. Cost Based (Price Optimization)

Selects the most cost-effective model based on token pricing:

• Evaluates input token pricing, tie-breaking by output token pricing, and finally by lowest active requests.
• Useful for batch processing, translation, and background summarization where cost efficiency is prioritized over latency.

Advanced Configurations

For custom deployments (or to understand the underlying configurations of the presets), Infralo exposes several engine-level parameters:

1. Retry Logic

Configures automated retries when downstream model providers return transient errors.

• Enable Automatic Retries: Master toggle to enable or disable request retries on failure.
• Max Attempts: Total number of execution attempts allowed (including the initial request). For example, setting Max Attempts = 2 allows 1 initial attempt + 1 retry.
• Per-Attempt Timeout: A dedicated deadline (in seconds) enforced for each individual LLM attempt. If a downstream provider does not respond within this window, the attempt is cancelled and evaluated against your configured retry conditions.
• Target Switching: When enabled, the load balancer automatically rotates to a different healthy target model within the deployment fleet for subsequent retry attempts.

Retry Conditions & Error Classification

Infralo classifies errors to determine whether a request should be retried or immediately returned to the client:

2. Circuit Breaker

Prevents degraded or overloaded model endpoints from choking the gateway.

• Failure Threshold: The number of consecutive failed requests (such as timeouts, 5xx errors, or rate limits) before Infralo "opens" the circuit breaker.
• Cooldown Duration: The amount of time (in seconds) the circuit remains open. While open, Infralo halts all traffic to the failing model and routes it to backup targets, letting the provider recover.

3. Rate Limit Buffers

Configures a safety margin below the provider's official rate limits to avoid triggering HTTP 429 (Too Many Requests) errors.

• RPM Buffer: The percentage of requests per minute (e.g. 0.10 for 10%) to hold back.
• TPM Buffer: The percentage of tokens per minute to hold back.

4. Latency Tracking

Configures how Infralo calculates provider performance to route around slow instances.

• EMA Alpha (α): The Exponential Moving Average coefficient (between 0 and 1). A higher alpha places more weight on recent request times, allowing the routing engine to react quickly to sudden provider slowdowns.

5. Response Cache

A built-in caching layer to store and reuse response payloads for identical prompt inputs.

• Cache Mode: Choose to inherit workspace caching defaults (Inherit) or define custom deployment rules (Custom).
• Enable Cache: Toggles key-value caching on or off.
• Time-to-Live (TTL): The duration (in seconds) that cached responses remain valid. Caching saves token costs and yields sub-millisecond response latencies.

6. Streaming Fallback Mode

Configures gateway behavior when client applications send stream=true requests to models that do not natively support SSE streaming or have POST-execution runtime modules attached.

• Streaming Mode Setting: Choose to inherit workspace settings (Inherit) or configure a deployment-level override (Custom).
• Auto Mode (Default): Executes upstream inference non-streamed behind the scenes and streams the final payload to the client using simulated Server-Sent Events (SSE) chunks.
• Strict Mode: Rejects non-streamable API requests with an HTTP 400 Bad Request error instead of simulating SSE streaming.

Deployment Overview & Live Monitoring

When you navigate to a specific deployment in your workspace, you are presented with the Deployment Overview dashboard. This page serves as a live control center, displaying real-time operational telemetry and an interactive lineage graph of your traffic distribution.

1. Live Runtime Status

When a deployment is active, the dashboard automatically polls the gateway for current performance telemetry:

• Auto-Refresh Interval: By default, the dashboard polls every 30 seconds. You can adjust this rate via the refresh dropdown to 10s, 15s, 30s, 1m, or 2m, or disable auto-refresh entirely. Click the manual Refresh button at any time to pull the latest state instantly.
• Disabled Banner: If a deployment is disabled, a status banner will alert you that traffic routing is suspended. Live polling and the lineage graph are deactivated until the deployment is re-enabled in settings.

2. Real-Time Telemetry Cards

The dashboard displays four aggregate live metric cards representing the health and request status of the deployment fleet:

• Deployment Health: Indicates overall status (Healthy, Degraded, or Unhealthy) based on the status of downstream target models.
• Active Requests: The total number of concurrent, in-flight API requests currently being processed by the target models.
• Avg Latency: The rolling Exponential Moving Average (EMA) latency across all functional target models in the deployment.
• Shared Consumers: The number of other virtual deployments that share this same fleet of target models.

3. Interactive LLM Lineage Graph

The dashboard displays an interactive, node-based flowchart mapping out how incoming requests flow through the virtualization layer to downstream model providers:

• Lineage Nodes: Represents the virtual deployment connection to individual target model nodes. Each target model node displays its provider logo (e.g. OpenAI, Google Gemini, Anthropic), name, and live performance metrics.
• Active Routing Weights: Connection lines display the load-balancing percentage currently routed to each model.
• Circuit Breaker Status: Badges on target model nodes indicate if the model is routing normally (status Closed) or if its circuit breaker has tripped (status Open) due to failures.
• Active request tracking: Individual model nodes display their local count of concurrent requests and average latency.