Knowledge base: RAG, parsers, GraphRAG
Collections, parsing, ingest, search vs query, GraphRAG, limits and the RAG quirks that bite in production.
Two ways in
| Who | Transport | Identity | Use |
|---|---|---|---|
| Backend / this MCP server | S2S REST + POST {s2s}/rag/mcp JSON-RPC with SHORT tool names |
service account (or user) token with rag scopes | all writes (create, parse, ingest, delete, build graph) + reads |
| Agent in a run | MCP gateway, FQN novaos.novaos-rag.<tool> |
run execution token + X-Tenant-ID |
reads only (search, list_documents, graph_query, global_query) |
S2S cuts calls at 30 s; the MCP gateway allows up to 600 s. Never send X-NovaOS-Tenant-Id/User-Id/Run-Id headers to S2S: identity comes from the JWT.
Tools and shapes
create_collection {name, description, visibility: tenant|private, embedding_provider, embedding_model, embedding_dimension, client_request_id}
list_collections {} | delete_collection {collection_id}
list_documents {collection_id} -> documents {document_id, filename, content_hash, chunk_count, truncation, created_at}
search {collection_id, query, top_k, filters?: {document_ids?, metadata?}, use_decomposition?, use_expansion?}
query {collection_id, question, top_k} | graph_query {collection_id, question} | global_query {collection_id, question}
build_graph, graph_status {collection_id} | get_job_status {job_id} | delete_document {collection_id, document_id} | list_document_chunks
REST: GET /rag/capabilities, POST /rag/parse {file:b64, filename, mode:"async", persist_artifact, parser?, parser_options?}, GET /rag/parse/jobs/{id}, POST /rag/ingest/from-parse {parse_artifact_id, collection_id, chunking_strategy, chunk_size, chunk_overlap, min_chunk_size, metadata?}.
Unknown arguments are silently ignored: validate your own requests. filters.document_ids (not top-level document_ids, which 400s); graph tools take question, not query.
Collections
- Org-wide and SHARED across apps:
list_collectionsshows every app's collections. Store your collection ids; never select by name. Never auto-create silently. - Embedding model is fixed per collection and must exist in the region: US
text-embedding-3-small/1536, EUtext-embedding-004/768, providerllm_gateway. A wrong model fails only at ingest/search. - Use
client_request_idfor idempotent create. A private collection created by a service account is closed to members.
Parsers (from live capabilities)
Sync: text csv html pymupdf pdfplumber camelot docx xlsx; async: docling surya anydoc. Preference ladder:
- PDF: docling (tables,
do_table_structure), pymupdf (digital PDFs, fast), surya/docling OCR for scans (do_ocr/force_ocr) - DOCX: docling, docx, anydoc | PPTX: docling, anydoc | XLSX: xlsx | CSV/TSV: csv | images: surya
- DOC/PPT/RTF/ODT/ODS/ODP/EPUB: anydoc |
.xls: no parser (convert) Sparse text (<40 chars/page) means re-parse with OCR. Never invent a parser not in capabilities.
Ingest pipeline that works
- Size: practical cap 7 MiB per upload (base64 JSON transport); split PDFs into page groups.
- Parse once with
persist_artifact; poll the parse job; keep the artifact id (expires after 7 days:ARTIFACT_NOT_FOUND-> re-parse once). ingest/from-parsewith chunking: tablestable_aware1024/64/40; pdf/docxsemantic512/64/50; elsesentence512/64/50 (sizes are CHARACTERS).- Persist the ingest
job_idimmediately; pollget_job_status(terminal: completed / failed / dead_letter / cancelled). A job instaging> 90 s is stuck: fail fast. - Never retry an ingest after a 504/timeout (it may already be accepted: duplicates).
- Duplicate-upload bug: a duplicate returns a NEW document id with 0 chunks. After ingest,
list_documentsand resolve bycontent_hash, then filename. - Metadata: values <= 256 chars, no empty values, ~20 keys; immutable after ingest.
- XLSX truncates at 500 rows; table chunks above ~400 rows fail: split sheets into 50-row bands with headers.
Tool:
novaos_kb_ingestimplements all of this.
Search vs query
search= hybrid vector + Postgres full-text, RRF (k=60), no rerank. Scores are fused ranks (~0.003-0.045, ~0.005 typical): flat and always top_k results even for unrelated queries. NEVER threshold; rerank (e.g. a flash model grading 0-3).- Planner OFF (
use_decomposition:false, use_expansion:false): faster, predictable; the planner drops the execution token/model and runs sequentially. query= plan + search + rerank + LLM answer with citations, fixed model, confidence floor 0.5, 504 at ~30 s on S2S. Avoid on latency paths.- The same chunk can appear twice: dedupe by chunk id / content hash.
- Keyword leg is weak on codes like
AB-123and splits non-ASCII words. - A failed search is not "no match": report search status; never claim documents lack a fact when no search ran.
GraphRAG
build_graph(a write, minutes) ->graph_status->graph_query(local, entity relationships) orglobal_query(corpus-wide). Build when most documents are indexed (e.g. >= 80%).- "GraphRAG not enabled" = feature off on the env (stop asking). A new ingest can make the graph stale/unqueryable until rebuilt.
Tool:
novaos_kb_graph.
Known RAG defects (observed)
Chunking config not applied (B7); chunk-size unit ambiguous (B8); confidence floor 0.5 (B15); job status timestamps/ids wrong (B16); no atomic replace (F5); no multi-collection search over MCP (F11); end users cannot shape chunk text before embedding; MCP errors can come back as HTTP 200 (R25); silent lexical-only fallback when query embedding fails (R18).