Designing Enterprise Document Q&A

1. Requirements

Functional

• Employees ask natural-language questions and get answers with citations to source documents
• Sources: Confluence, Google Drive, Slack (public channels plus the user's private ones), Jira
• Follow-up questions in a conversation
• Answers must only use content the asking user can access in the source system

Non-functional

• 50K employees, around 20M documents
• Changes searchable within 15 minutes. Deletions and permission changes within 15 minutes
• First token under 2 s at p95
• Auditable: who asked what, and which documents were used

2. Operating point

Quality and trust first. Wrong or leaked answers destroy adoption. Latency of 1–2 s to first token is fine for an internal tool, and cost is moderate at this scale.

3. Estimates

At 9 QPS, a hosted API is clearly the right call for generation. Embeddings can use an API or one small self-hosted GPU.

4. Architecture

Ingestion (offline)

A separate ACL sync service pulls permissions (document shares, space permissions, channel membership) and group memberships from each source and the identity provider, and updates chunk metadata.

Query (online)

5. Deep dives

Deep dive A: permissions

• Model: each chunk stores source, allowed_users, allowed_groups and visibility. Large groups are stored as group IDs, not expanded user lists.
• Query-time identity: resolve the user's groups (including nested groups) from the identity provider, cached for about 5 minutes, and add the filter visibility = public OR groups ∩ user_groups ≠ ∅ OR user ∈ allowed_users.
• Late-binding check for the final 8 chunks against the source system's permission API, where available. This catches ACL-sync lag for the few documents actually shown.
• Slack private channels and DMs: index them only for their members, or exclude DMs entirely by policy.
• Everything else is scoped too: caches keyed by permission scope, and traces restricted to authorised reviewers.

Deep dive B: freshness and deletes

• Webhooks give most updates within seconds. Sources without webhooks are polled with "modified since" every 5 minutes.
• Nightly reconciliation: list IDs per source, diff them against the index, and fix missing, stale and deleted documents.
• Versioned chunk replacement: a new version atomically replaces all of a document's chunks. Deletes tombstone immediately and compact later.
• Permission changes flow as high-priority events. They are more urgent than content edits.

Deep dive C: retrieval quality

• Parsing tuned per source: Confluence macros, Drive tables, Slack threads grouped into conversation chunks, Jira issue fields.
• Hybrid search (project codes, error messages and names need BM25), plus a reranker.
• Recency boost for fast-changing sources (Slack, Jira), and authority boost for official spaces.
• Eval set: 500 real questions collected with each department, labelled with the correct source documents. Track recall@20 for retrieval and answer correctness with an LLM judge on every change.

6. Evaluation and safety

• Faithfulness checks: verify citations exist in the provided context, and flag unsupported claims.
• "Don't know" behaviour: when rerank scores are low, say so and suggest where to look. Log these as content gaps.
• Prompt injection: documents are untrusted input. The assistant has no write tools, which limits the damage to wrong answers within the user's own access.
• Audit log: user, question, retrieved document IDs and answer, with retention per policy.

7. Bottlenecks and follow-ups