Global LLM Collection

            Provider
               ↓
     Global LLM Collection     ← You are here
               │ (Workspace Access)
               ▼
      Workspace LLM Access
        ├──────────────┐
        ▼              ▼
  Direct Model    Deployment
        \              /
         \            /
          ▼          ▼
         Virtual API Key
               ↓
        Gateway Request

The Global LLM Collection serves as the single source of truth for all machine learning models integrated into your Infralo environment. Platform administrators use the Global Collection to manage API credentials (see Virtual API Keys), track model pricing, and control global rate-limiting policies so the observability dashboard can compute cost reports.

Registering a Model

Administrators can register models in two ways:

1. Browse Catalog: Choose from pre-configured popular models (e.g. GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro). This pre-fills model capabilities, context windows, and default pricing.
2. Custom Registration: Manually specify provider endpoints, custom model aliases, and parameters. This is useful for private deployments, self-hosted models (e.g., Llama via vLLM), or custom Azure OpenAI resource groups.

Configuration Settings

The registration form contains several configuration groups:

1. General Info

• Alias Name: A unique identifier for the model across the platform (e.g., gpt-4o-production-us). Must consist of lowercase letters, digits, and hyphens. This is the identifier that workspaces and deployments reference.
• Provider: Select your provider vendor:
  • openai (OpenAI)
  • gemini (Google Gemini)
  • anthropic (Anthropic)
  • hosted_vllm (vLLM)
  • lm_studio (LM Studio)
  • custom (Custom OpenAI Compatible)
• Model Type: Select the modality. Options include chat (Completions) and embedding (Vector generation).
• Provider Model Name: The exact name recognized by the provider's API (e.g., gpt-4o, claude-3-5-sonnet-20240620).

2. API Configuration

Dynamic fields shown based on the requirements of the selected provider:

• API Key: API token to authenticate requests. Required for cloud providers (openai, gemini, anthropic). Can be left blank during edits to keep the existing key.
• API Base URL: Override the standard provider endpoint. Required for local or self-hosted providers (hosted_vllm, lm_studio, custom) (e.g., http://localhost:1234/v1 or http://10.0.0.5:8000/v1).
• API Version: Specify the API version query parameter, primarily used for specific custom gateways (e.g., 2024-02-15-preview).

3. Rate Limits

Set rate limits to prevent runaway costs or avoid tripping provider rate limits:

• RPM Limit: Requests Per Minute. Set to -1 for unlimited.
• TPM Limit: Tokens Per Minute. Set to -1 for unlimited.

4. Pricing & Context (Per 1 Million Tokens)

Specify pricing rates in USD per million tokens so that Infralo's observability dashboard can compute accurate cost reports:

• Max Input Tokens: The maximum prompt context length supported. Required for all model types.
• Max Output Tokens: The maximum response length supported. Required for chat type models.
• Cost per Input Token: The price for prompt/context tokens. Required for both types.
• Cost per Output Token: The price for completion/response tokens. Required for chat type models.
• Cost per Cached Input Token: The price for prompt tokens that hit provider-side context caches (e.g., Anthropic Prompt Caching or OpenAI Prompt Caching). Enabled only if the supports_prompt_caching capability is selected.
• Embedding Dimension: The size of output vector arrays. Required and only applicable for embedding type models (e.g., 1536).

5. Capabilities Checkboxes

Define model capabilities for routing engines and client SDKs to understand which features are supported. These are categorized into:

Core Capabilities

• Function Calling (supports_function_calling): Can process external functions/tools definition.
• Parallel Function Calling (supports_parallel_function_calling): Can execute multiple tools/functions in a single response turn (requires Function Calling).
• Tool Choice (supports_tool_choice): Supports forcing a specific tool or auto-selection (requires Function Calling).
• System Messages (supports_system_messages): Can parse and execute system instructions.
• Response Schema (supports_response_schema): Can guarantee response matches a custom structured JSON format.

Media Capabilities

• Vision (supports_vision): Can interpret images and multimodal visual inputs.
• Audio Input (supports_audio_input): Accepts raw audio formats.
• Audio Output (supports_audio_output): Generates audio responses.
• Video Input (supports_video_input): Accepts raw video formats.
• File Input (supports_pdf_input): Parses and reads files.

Advanced Capabilities

• Prompt Caching (supports_prompt_caching): Enables prompt context caching to reduce latency/cost.
• Web Search (supports_web_search): Natively queries the web for real-time information (requires Function Calling).
• Code Execution (supports_code_execution): Evaluates sandboxed code blocks (requires Function Calling).
• Computer Use (supports_computer_use): Controls UI/OS actions (requires Function Calling).
• File Search (supports_file_search): Retrieves documents from vector stores (requires Function Calling).
• URL Context (supports_url_context): Reads web page content from provided URLs.

Reasoning Capabilities

• Reasoning (supports_reasoning): Employs a thinking budget/reasoning process (e.g., o1, o3-mini, DeepSeek R1).
• Low Reasoning Effort (supports_low_reasoning_effort): Requests minimal thinking tokens (requires Reasoning).
• Max Reasoning Effort (supports_max_reasoning_effort): Requests maximum thinking tokens (requires Reasoning).
• Minimal Reasoning Effort (supports_minimal_reasoning_effort): Configures low reasoning constraints (requires Reasoning).
• None Reasoning Effort (supports_none_reasoning_effort): Disables reasoning tokens for a run (requires Reasoning).
• XHigh Reasoning Effort (supports_xhigh_reasoning_effort): Allocates maximum possible thinking budget (requires Reasoning).

Technical Capabilities

• Native Streaming (supports_native_streaming): Supports real-time token-by-token response streaming.
• Native Structured Output (supports_native_structured_output): Natively validates structured JSON outputs.
• Service Tier (supports_service_tier): Supports selecting higher latency/reliability routing tiers.
• Assistant Prefill (supports_assistant_prefill): Allows inserting starter tokens to steer the assistant's response.
• Preset (supports_preset): Supports predefined system presets and parameters.

6. Workspace Access

Control which workspaces can access this model. Workspace users can select this model for their playground or deployments only if workspace access has been granted by a platform administrator.

Testing Connections

Infralo includes a Test Connection feature directly on the registration form. Before saving, platform administrators can click Test Connection to send a minimal verification request (either aembedding or acompletion via LiteLLM) using the configured API credentials and endpoint.

On success, the latency is measured and reported in milliseconds. If connectivity fails, a descriptive error message from the provider is displayed, facilitating quick troubleshooting.