Field guide
Testing, evaluation, observability and metering
Local tests, eval suites, post-publish smoke tests, logs and traces, cost per run and org spending.
How NovaOS agents are tested
There is no local testing of NovaOS behavior. The execution token, LLM gateway, MCP gateway, RAG, runtime context and progress events exist only inside a NovaOS run. A local uvicorn run proves nothing about them, and the CLI sandbox needs a platform checkout. The proven loop (AssetNova post-publish ping suite, NovaDiag spike):
- Before publishing, local checks only:
- contract tests (
novaos_agent_test): manifest version == code, one skill, operations enum == code, uppercase statuses, nomemorypermission, input parsing - trap linter (
novaos_agent_lint), manifest validation (novaos_manifest_validate), agent card capture (novaos_agent_card: the module imports and skills match)
- contract tests (
- Publish -> platform-admin approval -> install (your own instance).
- Test ON NovaOS:
novaos_test_deployedrunseval/suite.jsoncases as real runs and checks status and answer (contains,exact_match,regex;expect_statusfor cases that must fail). It reports latency, cold vs warm start, cost and run ids. Start with ping (no model), one gateway model call, one clean failure; add a RAG question, a grounded question and a tool call as you use them. - For a backend: repeat the suite as the service account (
as: "<sa ref>") on the SA's own instance. - Fix -> bump version (manifest + AGENT_VERSION) -> publish -> approve -> reinstall -> test again.
- Debug failures with
novaos_run_diagnose(run, events, trace, agent logs for that run) andnovaos_agent_logs.
Notes:
- Keep a cheap no-model operation (
ping) for smoke runs; the first run after install is a cold start (30-70 s). - Count platform failures (429, cut streams, capacity) separately from quality failures.
- Probe agents pay off: a
modelsoperation listingavailable_models()shows what the gateway serves to THIS run. - Platform evals (
novaos-cli eval run/list) had gaps on some envs;novaos_test_deployedis the reliable path. - Record real runs and replay them as a demo safety net.
Logs and traces
novaos-cli agent logs <agent-id> [--run-id <run>] [--since 1h] [--limit 200] [-f](AGENT id, not instance id; empty is not failure).agent pod-logs <agent-id> --container agent|auth-proxy --tail-lines 100 --since-seconds 1200. Tools:novaos_agent_logs,novaos_run_diagnose.novaos-cli debug trace <run-id>,debug watch <run-id>,debug compare <a> <b>.- UI: Org Console > Build > agent > Logs > Execution Logs / Lifecycle Events; Runs (filter Status = Failed); Platform Admin > Workflows (Timeline, Input/Output); Audit Log with Entity Refs -> Entity Trail.
- Agents cannot reach Langfuse from FDE pods; the gateway logs with message logging off. Ship call records to your backend through an MCP tool if you need traces.
Cost and metering
- Per call:
x-litellm-response-costheader (non-streamed); streamed calls priced from tokens. Accumulate per turn: cost_usd, input/output tokens, llm_calls, calls_without_cost, by_model; returnusagein the agent result and store it (e.g. aturn_usagetable). - Reference: flash ("Pulse") ~$0.01 per chat turn, pro ("Quasar") ~$0.09, grounded pro call ~$0.037.
- Platform cost API (verified on FDE US, admin role): marketplace
GET /api/admin/cost/summary?tenant_id&period(total, by category, top agents, top plugins/models),/api/admin/cost/breakdown?group_by=agent|operation_type|plugin,/api/admin/cost/records?agent_id&page,/api/admin/cost/operation/{requestId},/api/admin/agents-usage,/api/admin/service-accounts/{id}/usage; period7d,30d,YYYY-MM,all. Tool:novaos_usage. Non-admins: console pages. - UI: Org Console > Insights > Cost / Usage, per SA Service Accounts > account > Usage, budgets Manage > Budgets, platform Cost Explorer.
- Org rule (Orbitron): PRs and ClickUp updates include the agent-run cost from the usage output.