Bonus: Agentic RAG the Uber Way - Chunking, Query Optimiser, Source Identifier, Hybrid Retrieval, and LLM-as-Judge

The knowledge base is built once - chunk, embed, store in a vector database - and the query -> retrieve -> generate loop runs per question, unlike a custom GPT that re-reads the whole file. The storage-room analogy: one unsorted pile retrieves badly; drawers (embeddings) labelled by meaning hold files (chunks). Chunking choices: fixed length; sliding window with overlap for continuity; sentence-based; semantic, split by meaning with embeddings and recommended for ~90% of cases despite the compute; recursive, by headings then paragraphs then sentences.

PROCEDURE as told: replace a plain PDF loader (which drops headings, tables and the table of contents) with a structure-preserving one - Uber used a loader on Google's Python document API that recursively extracts paragraphs, tables and TOC; run an LLM pass that adds a summary, FAQs and keywords to each section; build an index of title, description, page number and metadata so retrieval consults a light index before reading full chunks. This is where the reported 60% improvement came from.

PROCEDURE with the worked example. Query Optimiser: rewrite the vague query using who is asking and when - 'certificate?' becomes 'When will I receive the certificate of this mastermind?'. Source Identifier: use keywords and metadata against the index to narrow ~2,000 chunks to about 100 candidates before any vector search. Both are agents added to a pipeline that used to be a single pass-through; the same idea adds agents at retrieval, post-processing and generation.

Four strategies: BM25 keyword matching (precise phrases), vector search (meaning), graph retrieval (entity relationships), hybrid (combine and rerank, top-k). Uber's fix was hybrid: vector returned 50 chunks, BM25 15, 65 combined; a post-processing agent removed duplicates to 35 and reordered by relevance to the optimised query so the most relevant reaches the model first; then generation ('you will receive the certificate in 48 hours only if you attend 80% live'). Vector-only retrieval was named as a root cause of the original failures.

Metrics: faithfulness (answer consistent with retrieved context), relevance (answer and context address the query), and system KPIs - latency, cost, throughput, recall, precision, MRR, DCG. PROCEDURE: subject-matter experts review 100 real query / answer cycles once to create a fixed golden dataset; thereafter an LLM-as-judge batch-evaluates new queries against it (daily, not per query) and its notes feed back into the system prompt. Reported result: ~27% more acceptable answers overall, the largest share from retrieval and indexing, enough to deploy to Uber's internal security and privacy support.

The trainer states plainly that the 'certificate?' walkthrough is his own illustration of Uber's published approach, not their system; the 100-query benchmark, the 27% and 60% figures and the chunk counts are attributed to Uber without a link on air. Treat the architecture as the lesson and the numbers as claims to check against Uber's engineering blog.