Donut Invoice Extractor
Project page for our local invoice pipeline built on Donut (Document understanding transformer) — an OCR-free model that reads document images end-to-end.
Paper takeaways
OCR-free Document Understanding Transformer (Kim et al.) introduces Donut: a Swin vision encoder plus a BART-style text decoder trained with cross-entropy. Instead of piping OCR tokens into a second model, Donut maps a document image directly to a structured token sequence (JSON-like tags).
- Avoids OCR runtime cost and language / layout lock-in of off-the-shelf OCR.
- Stops OCR mistakes from propagating into the understanding stage.
- Pre-training uses SynthDoG synthetic documents for language / domain flexibility.
- Strong results on VDU tasks (parse, classify, DocVQA) for both speed and accuracy.
Our setup
Fine-tuned checkpoint:
rhovhannisyan/dmr-invoice-extractor
(~777MB). Task prompt <s_cord-v2>. Fields include
invoice number, date, net / tax / gross amounts, and currency when the page is
classified as an Invoice (receipts often return Other).
Live Gradio hosting on free CPU requires
Hugging Face PRO.
Sources for a Gradio app are in this Space under
gradio/.
Run locally
pip install transformers torch pillow pdf2image sentencepiece
python - <<'PY'
from transformers import DonutProcessor, VisionEncoderDecoderModel
from PIL import Image
import torch, re
model_id = "rhovhannisyan/dmr-invoice-extractor"
processor = DonutProcessor.from_pretrained(model_id)
model = VisionEncoderDecoderModel.from_pretrained(model_id).eval()
image = Image.open("invoice.png").convert("RGB")
pixel_values = processor(image, return_tensors="pt").pixel_values
decoder_input_ids = processor.tokenizer("<s_cord-v2>", add_special_tokens=False, return_tensors="pt").input_ids
outs = model.generate(pixel_values, decoder_input_ids=decoder_input_ids, max_length=768, use_cache=False)
seq = processor.batch_decode(outs)[0]
seq = re.sub(r"<.*?>", "", seq.replace(processor.tokenizer.eos_token, ""), count=1).strip()
print(processor.token2json(seq))
PY