DocQuery: An AI processing pipeline for documents, with RAG and vector search

Exam prep material tends to teach retrieval-augmented generation one piece at a time. A chunking snippet here, an embeddings call there, a vector query in a notebook. DocQuery puts the whole path in one solution. You upload a PDF in the browser, watch it move through chunking and embedding, then ask a question and read the answer with citations back to the pages it came from. Documents land in Azure Blob Storage, a transactional outbox publishes DocumentUploaded to a Service Bus topic, a chunking worker splits the text with PdfPig under a real token budget, an embedding worker generates vectors with Azure OpenAI into pgvector, and a separate query API answers questions over them. Because it is a real system rather than a sample, it also carries the parts a sample skips, such as Clean Architecture with a CQRS split, Polly pipelines on every outbound dependency, Testcontainers-backed infrastructure tests, ADRs, and Bicep plus Kustomize to deploy it all to AKS with workload identity.

• The full RAG path in one clonable solution consisting of five hosts, one Aspire command, emulators for everything except Azure OpenAI (which has none).
• Chunking treated as a design decision with real token counting with the o200k_base encoding, paragraphs and lists kept whole where they fit, and chunks that never cross a page so every citation can name one.
• Answers are grounded and traceable. The chat model is told to use the retrieved passages only, and each [n] marker maps to a citation with document, page, excerpt and similarity score.
• Messages and database rows cannot drift apart. The domain events are written to an outbox in the same transaction and relayed to a Service Bus topic, where each worker's subscription filters on message Subject.