Technical magazine draft
Embeddings are useful. They let a system find passages that mean roughly the same thing as a user query, even when the words differ. But enterprise document intelligence breaks when we pretend that semantic similarity is the whole job.
The recent wave of RAG writing has made this point sharper: retrieval failure is usually not mysterious. It is predictable. Exact identifiers disappear into tokenization. Negation is blurred. Dates and thresholds are treated like surrounding prose. Abbreviations are invisible unless the corpus teaches the system that they matter. A question that looks simple to a person can ask for a combination of source, time, citation, and exclusion constraints that a single vector lookup was never designed to represent.
The Caveau Is a Data Pipeline First
caveauAI is not built as a chat box with a vector database attached. It is a controlled document pipeline. Raw material enters from uploaded files, URLs, website truth sources, scrapers, shared packages, and operational connectors. For Do Better Norge, the same architecture supports a legal corpus where a missed citation or wrong date is not a cosmetic problem. It changes what a user can safely rely on.
That is why the first engineering move is not prompt tuning. It is making the corpus searchable in the ways people actually ask questions. A PDF clause, a legal section, a GitHub source file, a Jira ticket, and an ERP record all need clean text, source metadata, tenant boundaries, classification, and repeatable indexing. Only then does generation become a grounded final step rather than a guess.
Where Plain Embeddings Fail
Dense vectors are excellent at fuzzy semantic recall. They are weak at things that are not fuzzy. In enterprise systems, the brittle cases are often the most important cases: identifiers, legal references, exclusions, thresholds, temporal filters, product codes, and organization-specific synonyms.
Identifier blindness
Section numbers, CELEX codes, court references, product codes, and clause IDs are not concepts. They are keys. We route them through chunk_identifiers before fuzzy retrieval.
Vocabulary mismatch
Users may ask for cancellation while the document says rescission, or ask for BV while the corpus says Barnevernet. corpus_keyword_dictionary gives experts an auditable synonym layer.
Negation and thresholds
Embeddings do not reliably distinguish "premium but not deductible" from "premium and deductible", or "after 2020" from "around 2020". QueryParser keeps these predicates structural.
Context at the wrong level
A single paragraph may answer a quote lookup, but a document-level question may need the whole document summary. Synthetic doc-card chunks make document-level intent retrievable.
The fix is not to abandon embeddings. The fix is to stop asking embeddings to do the work of identifiers, dictionaries, dates, filters, and graphs.
Enrichment Happens Before Retrieval
We are adding enrichment columns to client_documents so the corpus carries more than raw body text. DocumentEnricher.php runs a bounded gpt-4o-mini pass over each document and returns structured JSON: summary, keywords, aliases, identifiers, entities, and dates. That output is persisted for search and UI use instead of being lost inside a one-off answer prompt.
The summary and keyword set also feed a synthetic doc-card chunk. A normal body chunk is good for passage-level questions. The doc-card is better for document-level intent: "Which document talks about product offering qualification?" or "Find the policy about escalation." This is the place where enrichment helps retrieval without stuffing the full document into every query.
Indexes, Not One Index
Hybrid RAG is often described as vector plus keyword search. In practice, reliable enterprise retrieval needs more retrieval surfaces, each doing the job it is suited for. caveauAI's current rollout keeps those surfaces explicit and inspectable.
Record truth
client_documents and client_chunks keep text, status, classification, source, enrichment JSON, and fallback chunk content.
Semantic recall
Vectors and payloads support tenant-filtered similarity search, hybrid fusion, source metadata, identifiers, and doc-card chunks.
Literal recall
Sparse keyword search catches names, phrases, and operational terms that should not depend on embedding geometry alone.
Expert language
corpus_keyword_dictionary promotes discovered aliases into approved query expansion rules scoped by client and corpus.
Exact routing
chunk_identifiers indexes normalized citations, sections, cases, regulations, and codes for route-before-fuzzy lookup.
Document relationships
FalkorDB stores document-to-reference edges so search can boost materials connected through citations and co-citation paths.
Identifiers Get Their Own Route
Legal and operational corpora are full of exact references: section numbers, court references, CELEX codes, ISO standards, product codes, internal case IDs. These are not fuzzy concepts. The shared CitationExtractor normalizes them into {kind, normalized, raw} objects. The same extractor feeds the FalkorDB graph and the chunk_identifiers table, so graph edges and exact lookup do not drift apart.
At query time, QueryParser.php checks for identifiers before the normal fuzzy path completes. If a user asks for a specific section or case reference, matching chunks are injected at the top of the result set. Vector search can still add surrounding context, but it no longer gets the first chance to miss the key.
Query Parsing Turns Language Into Controls
A user query contains more than a topic. It may contain a date window, an exclusion, a magnitude, a jurisdiction, a document type, or an exact code. Embedding the whole sentence makes those constraints soft. caveauAI now separates them.
QueryParser.php extracts negation, magnitude, temporal, and identifier intents with conservative rules. The semantic topic can still go to vector and keyword retrieval, but the constraints become filters, boosts, and routing decisions. That means "after 2020" is not merely adjacent to a vector. It becomes a structural condition that can be audited.
Graph Search Adds Relationship Memory
Some answers are not located in one chunk. They live in relationships: this case cites that statute, this guidance implements that article, this document mentions the same authority as another document. The graph layer gives caveauAI a way to preserve those relationships instead of hoping they appear in the same top-k vector neighborhood.
During ingestion, populateGraph() creates document and reference nodes in FalkorDB and writes citation edges using the shared extractor. During search, graph boost can lift candidates that are connected to exact identifiers or co-cited material. This is especially valuable for Do Better Norge, where a question about Norwegian family law may need statute text, court interpretation, ECHR context, and guidance in the same evidence trail.
The Do Better Norge Legal Corpus
tools.dobetternorge.no sits on top of this same retrieval discipline. The corpus includes shared legal library material and personal corpus search paths. It is a high-stakes setting: the system should not treat "Barnevernet", "BV", "ECHR Article 8", a Norwegian section reference, and a court citation as interchangeable fog.
That is why the DBN path is being aligned with the Caveau improvements: citation tokenization, identifier routing in LegalTools.php and corpus-search.php, temporal handling, and evidence-first answer composition. The point is not to make the model sound more confident. The point is to make the retrieved evidence more appropriate before the model speaks.
Operational Rollout
The work is intentionally staged. Fresh ingestion can start producing richer metadata immediately, while existing corpora are backfilled in batches. The current implementation plan keeps each enhancement additive and degradable: if enrichment or graph write fails, core ingestion should still complete; if exact routing has no match, hybrid retrieval continues.
- Migrations 166-168 add enrichment columns, corpus_keyword_dictionary, and chunk_identifiers.
- CitationExtractor becomes the shared source of truth for regex identifiers used by both graph population and identifier indexing.
- DocumentEnricher.php is wired into ingestDocument() so summary, keywords, aliases, identifiers, entities, and dates are persisted.
- Qdrant payloads now carry richer document metadata, including chunk_kind, keywords, and identifiers.
- QueryParser.php supplies negation, numeric, temporal, and identifier intent for routing and post-retrieval filtering.
- The DBN tools path is being aligned so LegalTools.php and corpus-search.php benefit from citation tokenization and identifier routing.
- Backfill runs in batches and resumes safely so existing corpora can be enriched without stopping fresh ingestion.
- The web UI is being extended with dictionary review, persisted summaries/keywords, and retrieval profile toggles.
The final step is evaluation. We are running small-corpus backfills and a failure-mode eval set: exact citation queries, synonym gaps, negation, time filters, numeric thresholds, multi-hop legal references, and document-level questions. Every change should make those cases more traceable, not merely more impressive in a demo.
Image Prompts for the Article
These prompts are written for ChatGPT or Gemini image generation. They avoid fake readable legal text and are designed as editorial illustrations for a technical data science article.
Raw data enters the caveau
16:9 editorial technical illustration for Blue Note Logic caveauAI. A secure vault-like data pipeline receives many document types: PDF pages, DOCX sheets, web pages, source code cards, legal case bundles, Jira tickets, and ERP records. The documents flow into a dark navy and brass "caveau" vault with clean labels for provenance and tenant boundary. Sophisticated enterprise AI aesthetic, cinematic lighting, sharp typography-like shapes but no readable fake text, no logos except abstract Blue Note Logic color cues.
Chunking with inherited metadata
16:9 cutaway diagram, refined technical magazine style. A long enterprise document is sliced into clean semantic sections around headings and clauses. Each slice carries small metadata tags for source, classification, date, corpus, and breadcrumb. Use dark graphite, cobalt, warm gold, and white. Make the chunk boundaries crisp and meaningful, not random. No readable fake text.
Three retrieval surfaces
16:9 systems architecture illustration showing one query splitting into three coordinated retrieval paths: vector search as a geometric field of points, keyword search as a precise index grid, and graph search as connected document nodes. The paths recombine into ranked evidence cards. Blue Note Logic/caveauAI palette: deep navy, black, brass, cool cyan accents. No fake UI text.
Exact citation routing
16:9 editorial illustration. A legal identifier token like a section mark, case number, or regulation code becomes a bright key that bypasses a foggy semantic search cloud and unlocks an exact source passage. Show the bypass path clearly as "exact route" visually, but avoid readable fake legal text. Serious legal-tech tone, dark navy background, gold keyline accents.
Query parser separates constraints
16:9 technical concept art. A user question enters a parser and separates into four clean lanes: semantic topic, negation, time window, numeric threshold, and identifier. The semantic lane goes to vector search while the structured lanes become filters. Elegant magazine infographic style, glassy dark panels, gold and cyan signal lines, no fake readable text.
Graph search over legal relationships
16:9 graph-search illustration for legal document intelligence. Nodes represent cases, statutes, articles, guidance, and client documents connected by citation edges. A highlighted path travels from a user question through several source nodes to an evidence card. Use restrained Nordic legal-tech mood, deep blue, graphite, brass, and soft white. No fake readable legal text.
Cited answer with evidence trail
16:9 product/editorial hybrid image. A final AI answer appears beside a source evidence rail with numbered citation cards and highlighted source snippets, all abstracted so no fake text is readable. Show confidence through structure: source cards, provenance markers, audit trail, and a clear boundary around retrieved evidence. Blue Note Logic/caveauAI colors, premium technical magazine finish.