Document extraction: four main approaches with a 1000x cost difference

by James Fishwick
Summary

I looked at the four main ways to turn unstructured documents into structured data: full LLM inference, fine-tuned small models, template-based extraction, and cloud OCR services.

The cost difference is huge: template-based extraction costs $0.001 per document, while full LLM inference costs $...

I went down a rabbit hole comparing the four main ways to turn unstructured documents into structured data: full LLM inference, fine-tuned small models, template-based extraction, and cloud OCR services. The cost spread floored me. Template extraction runs about $0.001 per document. Full LLM inference runs $5 to $15. Same PDF, same fields, a 1000x+ difference depending on which hammer you grab.

Most companies treat every document the same, which is how they end up paying LLM prices for forms a regex could read. Classify upfront and you can cut costs by 85%+ while keeping the flexibility for the weird edge cases. More on that below.

First up, cloud OCR services (Azure Document Intelligence, AWS Textract, Google Document AI). Fully managed, pre-trained on common document types, and $1.50 per 1,000 pages for basic OCR. Great for an MVP. The benchmark numbers surprised me. Gemini 2.0 Pro hit 100% item extraction accuracy at $0.0045 per invoice, while AWS and Azure both charge $0.01 per invoice, and Azure's asynchronous processing delivers 87% cost savings, with 30 pages costing $0.045 async versus $0.30 synchronous. The catch is that per-page pricing adds up fast, and Azure's custom extraction models jump to $50 per 1,000 pages.

Fine-tuned small models are where it gets fun. A 7B model like Llama 3.1 or Mistral 7B costs $0.00368 per 1,000 tokens of inference once trained, and the training itself is absurdly cheap. Under $2 for QLoRA on A100s. Mistral 7B takes 46 minutes. LLaMA-3 8B hit 76.6% accuracy with zero fine-tuning, matching a fine-tuned LLaMA-2 70B, and after fine-tuning on a measly 861 samples, LLaMA-2 7B jumped from 47.6% to 61.5% accuracy with 47.78% fewer hallucinations. The real bill is hosting. Cloud GPUs run $288 to $530 a month for inference, so you break even against the GPT-4 API somewhere around a million documents a year.

Template-based extraction costs almost nothing per document. The tax is that you build the templates ahead of time, but for known formats the current tools hit F1 scores of 1.0 in under a second. PyMuPDF scored between 0.983 and 0.993 on government, legal, and finance documents, and Camelot pulled a 0.828 F1 on complicated government tenders, tables included. Structured documents process in 0.3 to 1.6 seconds where multimodal LLM approaches take 33.9 seconds, which makes the template path 54 times faster. Azure Document Intelligence needs just 3 training and 3 test documents to build a template model, and the first 10 hours of neural training are free.

And then there's full LLM inference (Claude 3.5 Sonnet, GPT-4o, Gemini 2.0 and Gemini 2.5), the expensive Swiss Army knife at $0.005-0.02 per typical invoice. It handles any format without training and adapts when layouts change, plus you get reasoning about context along for the ride. In production, Claude and GPT-4o land 92-95% accuracy on line items and 95-98% on invoice extraction. Claude answers in 200 to 300 milliseconds. GPT-4o takes anywhere from 1 to 30 seconds depending on complexity. Prompt caching knocks 90% off repeated content, batch API processing takes 50% off workloads that can wait, and with caching you're looking at $30 to $90 a month for 10,000 invoices on Claude versus $50 to $180 on GPT-4o.

The sorting hat

The October 2024 Hybrid OCR-LLM Framework study lands on the architecture I'd pick too: a classifier out front, routing each document to the cheapest tool that can handle it. Standard forms, about 60% of the pile, go to table-based extraction (F1=1.0, 0.3s latency). Semi-structured documents, another 30%, get PaddleOCR plus the table method (F1=0.997, 0.6s). The last 10%, the genuinely novel formats, fall through to a multimodal LLM (F1=0.999, 34s).

IF standard_form → Template (F1=1.0, 0.3s, $0.001)
ELIF semi_structured → Fine-tuned 7B (F1=0.997, 0.6s, $0.03)
ELSE → LLM fallback (F1=0.999, 34s, $10)

Routing this way takes the blended cost from $10 per document for pure LLM down to $1.50, an 85% drop, and anything weird still reaches the big model. Asian Paints cut processing time from 5 minutes to 30 seconds per document, 10 times faster, saving 192 person-hours a month and turning up $47,000 in vendor overcharges along the way. You can even classify on the filename. Lightweight classifiers hit 96.7% accuracy at 442x the speed of full content analysis, pushing 80%+ of documents through the fast paths before an expensive model ever wakes up.

So which approach, if you have to start somewhere? Volume decides most of it. Above 10,000 documents a month, fine-tune or use templates for your common types. Mistral 7B trains in 46 minutes for $1.46 on RunPod (less than a latte) and gets you 85% of GPT-4's accuracy at an eighth of the cost. Under that volume, cloud OCR wins on speed to ship, and Google throws in the first 1,000 documents free before charging $30 per 1,000 pages for custom extractors. If accuracy is the whole game, go templates plus rules. Azure will compose up to 500 trained models and retrain incrementally on whatever gets misclassified. And when your formats are all over the place, LLM extraction it is. Claude 3.5 Sonnet swallows 100-page PDFs up to 30MB in its 200K token context window with zero preprocessing.

The AP departments doing this well pay $2.78 per invoice against an industry average of $9.40, 78% cheaper and 82% faster than their competitors. The broader market is headed the same way, with document extraction projected to grow from $10.57 billion in 2025 to $66.68 billion by 2032, a 30.6% annual clip, and my read is that the growth comes from companies wiring up smart routing rather than pointing an expensive LLM at everything.

Tools and Resources

Open-source PDF parsing:

Fine-tuning frameworks:

Cloud platforms:

RAG frameworks:

Key research papers:

Official documentation:

til

Related Posts