NovaOS MCP
Field guide

LLM gateway and models

Calling models through NovaOS: model ids, scopes, fallbacks, cost headers, rate limits, grounded web search, thinking tokens, and the prompt-injection filter.

Basics

  • OpenAI-compatible (LiteLLM) at LLM_GATEWAY_URL (+/v1) inside pods. Only the per-run execution token works: an app backend outside the cluster cannot call the LLM gateway directly; route model calls through an agent run (e.g. a complete skill).
  • LLMClient(base_url, execution_context=ctx.to_execution_context()) (extra [llm]), max_retries=0; await llm.available_models() lists what THIS run may use. From outside: GET {gateway}/api/v1/llm/models (developer role; tool novaos_llm_models).
  • Model ids have no vendor prefix: gemini-2.5-flash, gemini-2.5-pro, gpt-4o-mini (not google/...).
  • Access needs llm:model:<id> or llm:model:* on the CALLER token, and manifest permissions.tools including llm (and models: ["llm:model:*"]). Else MODEL_UNAUTHORIZED / MODEL_PREFLIGHT_FAILED.
  • Availability differs per env: FDE US served gemini-2.5-flash/-pro (Nov 2026); gemini-3.7/3.8-flash were not served; EU staging had no Claude Sonnet deployments (429) and serves gemini-2.5-pro as vertex_ai/gemini-2.5-pro. Probe before relying on a model.
  • Direct provider calls (own key as an installation secret) bypass NovaOS budgets and governance: avoid unless required.

Always

extra_body = {"disable_fallbacks": True}          # otherwise the gateway may silently swap models on 429
raw = await client.chat.completions.with_raw_response.create(model=m, messages=msgs, extra_body=extra_body)
served = raw.headers["x-litellm-model-group"]      # compare with m (tolerate vertex_ai/ prefix, -001 suffix)
cost = float(raw.headers.get("x-litellm-response-cost") or 0)

Streamed calls return no cost header: send stream_options={"include_usage": True} and price from tokens (pro $1.25/$10 per 1M in/out, flash $0.30/$2.50). Report finish_reason so an empty answer says why.

Limits and retries

  • ~10 in-flight model requests per tenant (FDE US), shared by every run and user: 429 / "Concurrent request limit": honor Retry-After, else 2 s exponential with jitter, 3 retries, max 15 s.
  • Limiter trouble ("rate_limit_unavailable", "redis error", 503): retry with backoff.
  • max_tokens reserves budget (held against SA budgets) but Gemini 2.5 thinking tokens count against it and truncate JSON: size it generously or omit on JSON steps. Thinking budget goes in extra_body.thinking = {"type":"enabled","budget_tokens":N}. Retry without reasoning_effort if rejected.
  • 400 Request blocked: potential prompt injection detected: the gateway filter matched prompt text (e.g. quoted "ignore previous instructions" or <<<NAME ... NAME>>> markers). Reword; do not retry.

Grounded web search (Gemini)

extra_body={"tools":[{"googleSearch":{}}], "disable_fallbacks": True} on gemini-2.5-pro. Sources in vertex_ai_grounding_metadata (groundingChunks[].web.{uri,title}); URIs are vertexaisearch.cloud.google.com/grounding-api-redirect/...: resolve before showing. ~$0.037 per call; can take 100+ s.

Embeddings

/v1/embeddings through the gateway from the agent (provider llm_gateway): US text-embedding-3-small (1536), EU text-embedding-004 (768).

Budgets

Org budgets: Org Console > Manage > Budgets (hard limit $/month per agent, resets on the 1st UTC; "Not yet enforced" defaults do not block). Usage: Insights > Cost / Usage; platform admins Platform Admin > Cost Explorer.