RAG Pipeline for Compliance Document Processing

Representative project based on real engagements. Client details anonymized.

The situation

A financial compliance team processed 200+ regulatory documents per week. Each document was a PDF with inconsistent formatting — some were scanned images, some were structured forms, some were free-text emails. Analysts spent 45 minutes per document extracting key fields (company name, regulatory reference, deadline, risk classification) and entering them into a spreadsheet.

The team had 3 analysts spending 60% of their time on data extraction. Errors were common because the work was tedious and the documents were inconsistent.

What we built

Document ingestion (week 1): A FastAPI service accepts PDF uploads. Scanned documents go through Tesseract OCR. Native PDFs are parsed with PyPDF2. The raw text is cleaned and normalized.

RAG pipeline (week 2): Documents are chunked into 500-token segments and embedded using OpenAI's text-embedding-ada-002 model. Embeddings are stored in Pinecone. When an analyst requests extraction, the system retrieves the most relevant chunks and sends them to GPT-4 with a structured prompt that specifies the exact fields to extract.

Validation layer (week 3): Extracted data is validated against a PostgreSQL knowledge base of known regulatory references, company names, and date formats. Fields that fail validation are flagged with confidence scores. Low-confidence extractions (< 85%) go to human review with the source text highlighted.

Review interface (week 4): A simple React interface shows extracted fields side-by-side with the source document. Analysts can approve, edit, or reject each field. Approved extractions are pushed to their existing spreadsheet via API.

The numbers

Metric	Before	After
Processing time per document	45 minutes	3 minutes (automated) + 2 minutes (human review)
Weekly processing capacity	200 documents	800+ documents
Extraction accuracy	~90% (human, inconsistent)	97.2% (automated + human review)
Analyst time on data entry	60% of their week	15% of their week
Cost per document	~$22 (analyst time)	~$1.50 (API + minimal review)

What we learned

The biggest challenge was prompt engineering. Generic prompts like "extract key information" produced inconsistent results. We had to build document-type-specific prompts with explicit field definitions, example outputs, and fallback instructions for missing fields. The validation layer was critical — without it, the 97.2% accuracy would drop to ~85%, which is not acceptable for compliance work.

The other lesson: do not try to replace the analysts. The system handles the extraction; the analysts handle the judgment calls. This framing made the team adopt the tool instead of resisting it.