NovaOS MCP
Field guide

Testing, evaluation, observability and metering

Local tests, eval suites, post-publish smoke tests, logs and traces, cost per run and org spending.

How NovaOS agents are tested

There is no local testing of NovaOS behavior. The execution token, LLM gateway, MCP gateway, RAG, runtime context and progress events exist only inside a NovaOS run. A local uvicorn run proves nothing about them, and the CLI sandbox needs a platform checkout. The proven loop (AssetNova post-publish ping suite, NovaDiag spike):

  1. Before publishing, local checks only:
    • contract tests (novaos_agent_test): manifest version == code, one skill, operations enum == code, uppercase statuses, no memory permission, input parsing
    • trap linter (novaos_agent_lint), manifest validation (novaos_manifest_validate), agent card capture (novaos_agent_card: the module imports and skills match)
  2. Publish -> platform-admin approval -> install (your own instance).
  3. Test ON NovaOS: novaos_test_deployed runs eval/suite.json cases as real runs and checks status and answer (contains, exact_match, regex; expect_status for cases that must fail). It reports latency, cold vs warm start, cost and run ids. Start with ping (no model), one gateway model call, one clean failure; add a RAG question, a grounded question and a tool call as you use them.
  4. For a backend: repeat the suite as the service account (as: "<sa ref>") on the SA's own instance.
  5. Fix -> bump version (manifest + AGENT_VERSION) -> publish -> approve -> reinstall -> test again.
  6. Debug failures with novaos_run_diagnose (run, events, trace, agent logs for that run) and novaos_agent_logs.

Notes:

  • Keep a cheap no-model operation (ping) for smoke runs; the first run after install is a cold start (30-70 s).
  • Count platform failures (429, cut streams, capacity) separately from quality failures.
  • Probe agents pay off: a models operation listing available_models() shows what the gateway serves to THIS run.
  • Platform evals (novaos-cli eval run/list) had gaps on some envs; novaos_test_deployed is the reliable path.
  • Record real runs and replay them as a demo safety net.

Logs and traces

  • novaos-cli agent logs <agent-id> [--run-id <run>] [--since 1h] [--limit 200] [-f] (AGENT id, not instance id; empty is not failure). agent pod-logs <agent-id> --container agent|auth-proxy --tail-lines 100 --since-seconds 1200. Tools: novaos_agent_logs, novaos_run_diagnose.
  • novaos-cli debug trace <run-id>, debug watch <run-id>, debug compare <a> <b>.
  • UI: Org Console > Build > agent > Logs > Execution Logs / Lifecycle Events; Runs (filter Status = Failed); Platform Admin > Workflows (Timeline, Input/Output); Audit Log with Entity Refs -> Entity Trail.
  • Agents cannot reach Langfuse from FDE pods; the gateway logs with message logging off. Ship call records to your backend through an MCP tool if you need traces.

Cost and metering

  • Per call: x-litellm-response-cost header (non-streamed); streamed calls priced from tokens. Accumulate per turn: cost_usd, input/output tokens, llm_calls, calls_without_cost, by_model; return usage in the agent result and store it (e.g. a turn_usage table).
  • Reference: flash ("Pulse") ~$0.01 per chat turn, pro ("Quasar") ~$0.09, grounded pro call ~$0.037.
  • Platform cost API (verified on FDE US, admin role): marketplace GET /api/admin/cost/summary?tenant_id&period (total, by category, top agents, top plugins/models), /api/admin/cost/breakdown?group_by=agent|operation_type|plugin, /api/admin/cost/records?agent_id&page, /api/admin/cost/operation/{requestId}, /api/admin/agents-usage, /api/admin/service-accounts/{id}/usage; period 7d, 30d, YYYY-MM, all. Tool: novaos_usage. Non-admins: console pages.
  • UI: Org Console > Insights > Cost / Usage, per SA Service Accounts > account > Usage, budgets Manage > Budgets, platform Cost Explorer.
  • Org rule (Orbitron): PRs and ClickUp updates include the agent-run cost from the usage output.