Designing Enterprise Document Q&A
A full walkthrough for an internal assistant that answers questions over Confluence, Drive, Slack and Jira for 50K employees, covering connectors, parsing and chunking, permissions, freshness, hybrid retrieval, reranking, grounded answers with citations, evals and security.
"Ask the company anything" is the most common enterprise GenAI project, and the one most often cancelled after the pilot. The model is rarely the reason. Pilots fail because answers cite a stale page, miss the one Slack thread that mattered, or, worst of all, reveal a document the user was never allowed to see. This design treats RAG as a search and data-governance system with an LLM at the end.
1. Clarify the task and the quality bar
Questions to ask:
- Which sources, and how big? Assume Confluence, Google Drive, Slack and Jira: about 20M documents in total for 50K employees.
- Whose permissions apply? The asking user's, exactly as in the source system. This is non-negotiable.
- How fresh? Edits, deletions and permission changes should be reflected within 15 minutes.
- Answer or search? Both: a synthesised answer with citations, plus the underlying sources so users can verify.
- Any actions? No. Read-only for v1 (no "create a Jira ticket"), which shrinks the security surface a lot.
- What does a good answer look like? Correct, grounded only in retrieved content, cited, and honest when the answer isn't in the corpus.
Functional requirements
- Natural-language questions with streamed answers and inline citations
- Follow-up questions in a conversation
- Filters: source, space or channel, date range, author
- "Show me the sources" view, and feedback on each answer
- Admin controls: which sources and spaces are indexed, and exclusions (HR, legal holds)
Non-functional requirements
- 50K employees, ≈ 20M documents, ≈ 250K questions a day
- Freshness: 15 minutes for content, deletions and permission changes
- TTFT under 2 s at p95
- Zero tolerance for permission leaks; full audit trail
- Data residency: everything stays in the company's cloud tenancy and region
2. Choose the operating point
Quality and trust first. A wrong answer delivered confidently, with a citation, is worse than no answer. A leaked answer ends the project. Latency of 1–2 s to first token is fine for an internal tool, and at under 10 QPS the cost is moderate. So we spend budget on retrieval quality (reranking, better parsing) and on safety checks, not on shaving milliseconds.
3. Estimates
| Quantity | Value |
|---|---|
| Questions per day | 50K × 5 = 250K |
| Peak QPS (work hours, 3×) | ≈ 9 QPS |
| Chunks | 20M docs × ≈ 5 chunks = 100M chunks |
| Vector index (1,024-dim int8, HNSW M = 16) | 100M × (1 KB + ≈ 140 B) ≈ 115 GB per replica |
| BM25 index | ≈ 50–100 GB (100M chunks × ≈ 500 tokens, compressed) |
| Initial embedding backfill | 100M × 500 tokens = 50B tokens ≈ $1K at $0.02 per M tokens |
| Daily changes (≈ 2% of docs) | 400K docs, so ≈ 2M chunks re-embedded a day (≈ $20) |
| Reranking | 9 QPS × 100 candidates = 900 pairs/s: one GPU |
| Tokens per question | ≈ 6K in (instructions + 8 chunks + history), 400 out |
| LLM cost ($2 / $10 per M tokens) | 250K × ($0.012 + $0.004) ≈ $4K a day, ≈ $120K a month before caching |
At 9 QPS, a hosted API (inside the company's cloud tenancy) is clearly right for generation. Embeddings and reranking can be an API or a single small self-hosted GPU. The index fits on a few nodes. None of this is a capacity problem, so the interview time belongs to permissions, freshness and quality.
4. Architecture
Two pipelines share one index: ingestion keeps it current, and the query path reads it.
Ingestion (offline)
A separate ACL sync service pulls permissions (document shares, space permissions, channel membership) and group memberships from each source and from the identity provider, and updates chunk metadata without re-embedding.
Query (online)
Components
| Component | Responsibility |
|---|---|
| Connectors | One per source: webhooks, incremental polling, backfill, rate-limit handling |
| Change queue | Durable per-document events with priorities (permission > delete > edit > new) |
| Parser / chunker | Source-specific extraction into structured chunks |
| Embedding service | Batched embeddings, with the model version stored on every vector |
| Hybrid index | Vector (HNSW) + BM25 with metadata and ACL fields, in one engine such as OpenSearch or Vespa, or two engines fused |
| ACL sync | Source permissions and identity-provider groups into chunk metadata |
| Query service | Identity, query rewriting, retrieval, reranking, prompt assembly, generation, citation checks |
| Conversation store | Turns and the sources used, for follow-ups and audit |
| Eval and analytics | Golden set runs, feedback, "no answer" content gaps |
Data model (per chunk)
chunk_id, doc_id, doc_version, source, url, title, section_path,
text, embedding(model_version), updated_at, author,
visibility, allowed_users[], allowed_groups[], space_id5. Deep dives
Deep dive A: permissions
Permissions are the deep dive to open with, before the interviewer asks.
- Model: each chunk stores
visibility,allowed_usersandallowed_groups. Large groups stay as group IDs; they are not expanded into user lists, which would make every group change a million-row update. - Query-time identity: resolve the user's groups, including nested ones, from the identity provider (Okta, Entra ID), cached for about 5 minutes. Apply the filter
visibility = public OR groups ∩ user_groups ≠ ∅ OR user ∈ allowed_usersinside the vector search, not after it. Post-filtering the top 100 can leave nothing, or leave only documents that barely match. - Late-binding check: for the final 8 chunks, ask the source system's permission API whether the user can still see each document, where the source offers one. This closes the ACL-sync lag gap for exactly the documents that will be shown, at the cost of ~50–100 ms.
- Source quirks: Drive has "anyone with the link" sharing (treat as not public), Confluence has space and page restrictions that combine, and Slack private channels and DMs should be indexed only for members or excluded entirely by policy.
- Everything derived is scoped too. Caches are keyed by permission scope, conversation history is per user, and traces containing document text are visible only to authorised reviewers.
More patterns in Permissions and multi-tenancy.
Deep dive B: connectors, freshness and deletes
- Webhooks first. Confluence, Drive, Slack and Jira all offer change events, which give most updates within seconds. Sources without them are polled with "modified since" every few minutes.
- Respect source rate limits. An initial backfill of millions of Drive files can take days under API quotas. Plan it as a throttled, resumable job, and prioritise recently active content so the product is useful early.
- Nightly reconciliation: list IDs per source, diff them against the index, and repair missing, stale and deleted documents. Events get lost; reconciliation is the safety net.
- Versioned chunk replacement: a new document version atomically replaces all of its chunks (delete by
doc_idand old version, then insert), so a shrinking document never leaves orphan chunks behind. - Priorities: permission changes and deletions outrank content edits. A revoked share must disappear from answers before a typo fix appears.
- Freshness SLO dashboards per connector: lag from source change to searchable, with alerts. See Freshness, deletes and re-indexing.
Deep dive C: parsing and chunking
Most retrieval failures trace back to bad chunks, not bad embeddings.
- Source-aware parsing. Confluence pages keep their heading hierarchy; macros and tables are rendered to text; code blocks are kept whole. Drive PDFs need layout-aware extraction and OCR for scans; spreadsheets become row groups with headers repeated. Slack messages are grouped by thread, because a single message ("yes, do that") is meaningless alone. Jira issues combine title, description, status and key comments.
- Structure-aware chunks of ≈ 300–600 tokens that respect headings, with a small overlap.
- Contextual headers: prepend each chunk with its document title and section path ("Payments runbook › Refunds › Manual override") before embedding. This cheap trick fixes many "the chunk doesn't say what it's about" misses.
- Parent-child retrieval: search small chunks for precision, then give the model the surrounding section for context.
Chunking playground
Split a refund policy three ways. The question to answer later: "When does the refund window start for enterprise customers?"
- Chunk 1≈ 48 tokens
# Refund policy ## Eligibility Customers on annual plans can request a full refund within 30 days of purchase. Monthly plans are not refundable, but you can cancel at any time and keep access
- Chunk 2≈ 48 tokens
until the end of the billing period. ## Enterprise contracts Enterprise customers follow the terms in their signed order form. If the order form is silent on refunds, the standard 30-day win
- Chunk 3≈ 48 tokens
dow applies from the contract start date, not the invoice date. ## How to request Open a ticket from the billing page and choose "Refund request". Refunds are issued to the original payment m
- Chunk 4≈ 48 tokens
ethod within 5–10 business days. Bank transfers can take up to 15 business days. ## Exceptions We do not refund add-on usage such as extra API calls or storage overages, even within the 30-da
- Chunk 5≈ 3 tokens
y window.
Fixed-size chunks cut sentences in half and separate facts from the heading that gives them meaning. Structure-aware chunking keeps sentences whole and repeats the section title in each chunk, so "30 days from the contract start date" stays attached to "Enterprise contracts".
Deep dive D: retrieval quality
- Query rewriting. Follow-ups ("what about for contractors?") are rewritten into standalone questions using the conversation. A small model also extracts filters ("in the last quarter", "in the payments space").
- Hybrid search. Dense vectors catch paraphrases; BM25 catches project codes, error messages, acronyms and people's names, which embeddings blur. Fuse the two lists with reciprocal rank fusion (RRF). See Hybrid search and reranking.
- Reranking. A cross-encoder scores the top 100 candidates against the question and keeps the best 8. It is usually the single biggest quality gain after good chunking, for ≈ 100–200 ms.
- Boosts: recency for fast-changing sources (Slack, Jira), authority for official spaces, and a penalty for archived or deprecated pages.
- Diversity: avoid 8 chunks from the same document when the question spans several.
- Embedding upgrades are migrations: store the model version on every vector, build the new index in parallel, compare on the eval set, then switch.
Deep dive E: grounded generation and citations
The generation prompt instructs the model to answer only from the provided sources, cite each claim with a source ID, and say "I couldn't find this" when the sources don't cover the question.
- Chunks are delimited and labelled (
<source id="S3" title="…" url="…">…</source>), so citations map back to real documents. - Citation verification: after generation, check that every cited ID was in the context and that cited passages support the claim (a fast NLI model or a small LLM check). Unsupported sentences are flagged or removed.
- "Don't know" behaviour: when rerank scores are all low, skip generation and show the best matches with "I couldn't find a definitive answer". These questions are logged as content gaps for documentation owners, which becomes one of the product's most valued outputs.
- Conflicting sources: when two documents disagree, show both, with dates, rather than picking one silently.
Deep dive F: security
- Prompt injection. Documents are untrusted input: anyone who can edit a wiki page can plant instructions. The assistant has no write tools, so the worst case is a wrong answer within the user's own access. Still, strip or neutralise instruction-like content in the system prompt design, and never let retrieved text change tool permissions. See Prompt injection and guardrails.
- Exfiltration via rendering. A planted instruction can ask the model to output a markdown image whose URL contains data (
). Disable remote image rendering in answers, or allow-list domains. - Sensitive content. Exclude HR, legal-hold and credential-bearing spaces at the connector level, and run secret and PII detectors during ingestion.
- Audit log: user, question, retrieved document IDs, answer and citations, retained per policy.
6. Evaluate, safeguard, operate
Evaluation has to separate retrieval from generation, or you can't tell which one to fix.
| Layer | Metric | How |
|---|---|---|
| Retrieval | Recall@20, MRR | 500+ real questions collected with each department, labelled with the correct source documents |
| Reranking | Precision@8 | Same set |
| Answers | Correctness, faithfulness, citation accuracy | LLM-as-judge calibrated on a few hundred human-graded answers |
| Behaviour | "Don't know" precision | Questions with no answer in the corpus must get "I couldn't find it" |
| Security | Leak rate (must be 0) | Synthetic users with restricted access asking about restricted documents |
Run the suite on every change to parsing, chunking, embeddings, retrieval or prompts. Online: thumbs, citation click-through, "no answer" rate by department, and reformulation rate (users rephrasing the same question is a quiet failure signal). See Offline evals and golden sets.
Observability. Trace each question through rewrite, retrieval (with scores), rerank, generation and citation checks. The most useful debugging view shows the retrieved chunks next to the answer.
Failure modes
| Failure | Mitigation |
|---|---|
| ACL sync falls behind | Late-binding check, priority queue for permission events, alert on reconciliation diffs |
| A connector silently stops | Freshness-lag alerts per connector; reconciliation catches it within a day |
| The LLM API is slow or down | Fall back to a second provider or model; degrade to search results with no answer |
| Index node loss | Replicas; the index is rebuildable from the sources and the embedding cache |
| Bad parser release | Golden-set gate in CI; re-parse only the affected source |
7. Cost and how the design evolves
Cost. Generation dominates at ≈ $120K a month. Levers: a smaller model for simple questions (routing), fewer and tighter chunks (8 → 5 with a better reranker), prompt caching of the instructions, and permission-scoped caching of retrieval results for popular questions. Indexing and reranking cost a few thousand dollars a month.
Evolution.
- Pilot: two sources, one department, hybrid search plus an API model, and a 200-question eval set.
- Company-wide: all sources, the ACL sync service, late-binding checks, a reranker, and content-gap reporting.
- Next: agentic retrieval for multi-hop questions ("which teams changed their on-call process after the March incident?"), structured sources (text-to-SQL over HR or finance data with its own permission model), and carefully permissioned actions.
8. Bottlenecks and follow-ups
| Likely question | Answer sketch |
|---|---|
| "A user got an answer from a doc they can't see?" | ACL-sync lag. Late-binding check on final chunks, prioritise permission events, alert on reconciliation diffs, and audit the incident from the log |
| "Answers are stale" | Measure freshness lag per connector, fix polling gaps, add a recency boost and show document dates in citations |
| "Scale to 500K employees?" | The index grows about 10× (shard it by source or business unit); QPS to about 90. Still API-friendly |
| "Multi-hop questions?" | An agentic retrieval path with a step budget, used only when the router detects it |
| "How do you know it's getting better?" | Separate retrieval and answer metrics on a golden set, gated in CI, plus online reformulation and "no answer" rates |
| "Why not fine-tune on the documents?" | Fine-tuning doesn't respect permissions, goes stale, and can't cite. See RAG vs fine-tuning vs long context |
Key takeaways
- At this scale the hard problems are permissions, freshness and retrieval quality, not GPU capacity.
- Sync ACLs from every source, attach them to every chunk, filter retrieval by the caller's identity and groups, and re-check the final chunks against the source.
- Event-driven connectors plus a reconciliation job keep the index within minutes of the sources, including deletes and permission changes.
- Quality comes from the pipeline, in order of impact - parsing and chunking, hybrid retrieval, reranking, then the generation prompt.
- Measure retrieval and answers separately on a labelled question set, and treat every document as untrusted input.