Supercharge your OCR Pipelines with Open Models
We have added Chandra and OlmOCR-2 to this blog, as well as OlmOCR Scores of the models 🫡
TL;DR: The rise of powerful vision-language models has transformed document AI. Each model comes with unique strengths, making it tricky to choose the right one. Open-weight models offer better cost efficiency and privacy. To help you get started with them, we’ve put together this guide.
In this guide, you’ll learn:
The landscape of current models and their capabilities
When to fine-tune models vs. use models out-of-the-box
Key factors to consider when selecting a model for your use case
How to move beyond OCR with multimodal retrieval and document QA
By the end, you’ll know how to choose the right OCR model, start building with it, and gain deeper insights into document AI. Let’s go!
Table-of-Contents
Brief Introduction to Modern OCR
Optical Character Recognition (OCR) is one of the earliest and longest running challenges in computer vision. Many of AI’s first practical applications focused on turning printed text into digital form.
With the surge of vision-language models (VLMs), OCR has advanced significantly. Recently, many OCR models have been developed by fine-tuning existing VLMs. But today’s capabilities extend far beyond OCR: you can retrieve documents by query or answer questions about them directly. Thanks to stronger vision features, these models can also handle low-quality scans, interpret complex elements like tables, charts, and images, and fuse text with visuals to answer open-ended questions across documents.
Model Capabilities
Transcription
Recent models transcribe texts into a machine-readable format.
The input can include:
Handwritten text
Various scripts like Latin, Arabic, and Japanese characters
Mathematical expressions
Chemical formulas
Image/Layout/Page number tags
OCR models convert them into machine-readable text that comes in many different formats like HTML, Markdown and more.
Handling complex components in documents
On top of text, some models can also recognize:
Images
Charts
Tables
Some models know where images are inside the document, extract their coordinates, and insert them appropriately between texts. Other models generate captions for images and insert them where they appear. This is especially useful if you are feeding the machine-readable output into an LLM. Example models are OlmOCR by AllenAI, or PaddleOCR-VL by PaddlePaddle.
Models use different machine-readable output formats, such as DocTags, HTML or Markdown (explained in the next section Output Formats). The way a model handles tables and charts often depends on the output format they are using. Some models treat charts like images: they are kept as is. Other models convert charts into markdown tables or JSON, e.g., a bar chart can be converted as follows.
Similarly for tables, cells are converted into a machine-readable format while retaining context from headings and columns.
Output formats
Different OCR models have different output formats. Briefly, here are the common output formats used by modern models.
DocTag: DocTag is an XML-like format for documents that expresses location, text format, component-level information, and more. Below is an illustration of a paper parsed into DocTags. This format is employed by the open Docling models.
HTML: HTML is one of the most popular output formats used for document parsing as it properly encodes structure and hierarchical information.
Markdown: Markdown is the most human-readable format. It’s simpler than HTML but not as expressive. For example, it can’t represent split-column tables.
JSON: JSON is not a format that models use for the entire output, but it can be used to represent information in tables or charts.
The right model depends on how you plan to use its outputs:
Digital reconstruction: To reconstruct documents digitally, choose a model with a layout-preserving format (e.g., DocTags or HTML).
LLM input or Q&A: If the use case involves passing outputs to LLM, pick a model that outputs Markdown and image captions, since they’re closer to natural language.
Programmatic use: If you want to pass your outputs to a program (like data analysis), opt for a model that generates structured outputs like JSON.
Locality Awareness
Documents can have complex structures, like multi-column text blocks and floating figures. Older OCR models handled these documents by detecting words and then the layout of pages manually in post-processing to have the text rendered in reading order, which is brittle. Modern OCR models, on the other hand, incorporate layout metadata to help preserve reading order and accuracy. This metadata is called “anchor”, it can come in bounding boxes. This process is also called as “grounding/anchoring” because it helps with reducing hallucination.
Model Prompting
OCR models can either take in images and an optional text prompt, this depends on the model architecture and the pre-training setup.
Some OCR models support prompt-based task switching, e.g. granite-docling can parse an entire page with the prompt “Convert this page to Docling” while it can also take prompts like “Convert this formula to LaTeX” along with a page full of formulas.
Other models, however, are trained only for parsing entire pages, and they are conditioned to do this through a system prompt.
For instance, OlmOCR by AllenAI takes a long conditioning prompt. Like many others, OlmOCR is technically an OCR fine-tuned version of a VLM (Qwen2.5VL in this case), so you can prompt for other tasks, but its performance will not be on par with the OCR capabilities.
Cutting-edge Open OCR Models
We’ve seen an incredible wave of new models this past year. Because so much work is happening in the open, these players build on and benefit from each other’s work. A great example is AllenAI’s release of OlmOCR, which not only released a model but also the dataset used to train it. With these, others can build upon them in new directions. The field is incredibly active, but it’s not always obvious which model to use.
Comparing Latest Models
To make things a bit easier, we’re putting together a non-exhaustive comparison of some of our current favorite models. All of the models below are layout-aware and can parse tables, charts, and math equations. The full list of languages each model supports are detailed in their model cards, so make sure to check them if you’re interested. All models below have open-source license except for Chandra having OpenRAIL license and Nanonets license being unclear. The average scores are taken from model cards of Chandra, OlmOCR, evaluated on OlmOCR Benchmark, which is English-only. Many of the models in this collection have been fine-tuned from Qwen2.5-VL or Qwen3-VL, so we also provide Qwen3-VL model below as well.
Model Name
Output formats
Features
Model Size
Multilingual?
Average Score on OlmOCR Benchmark
structured Markdown with semantic tagging (plus HTML tables, etc.)
Captions images in the documents
Signature & watermark extraction
Handles checkboxes, flowcharts, and handwriting
4B
✅Supports English, Chinese, French, Arabic and more.
N/A
Markdown, JSON, HTML tables and charts
Handles handwriting, old documents
Allows prompting
Converts tables & charts to HTML
Extracts and inserts images directly
0.9B
✅Supports 109 languages
N/A
Markdown, JSON
Grounding
Extracts and inserts images
Handles handwriting
3B
✅Multilingual with language info not available
79.1 ± 1.0
Markdown, HTML, LaTeX
Grounding
Optimized for large-scale batch processing
8B
❎English-only
82.3 ± 1.1
DocTags
Prompt-based task switching
Ability to prompt element locations with location tokens
Rich output
258M
✅Supports English, Japanese, Arabic and Chinese.
N/A
Markdown, HTML
Supports general visual understanding
Can parse and re-render all charts, tables, and more into HTML
Handles handwriting
Memory-efficient, solves text through image
3B
✅Supports nearly 100 languages
75.4 ± 1.0
Markdown, HTML, JSON
Grounding
Extracts and inserts images as is
9B
✅Supports 40+ languages
83.1 ± 0.9
Vision Language Model can output in all formats
Can recognize ancient text
Handles handwriting
Extracts and inserts images as is
9B
✅Supports 32 languages
N/A
While Qwen3-VL itself is a powerful and versatile vision-language model post-trained for document understanding and other tasks, it isn’t optimized for a single, universal OCR prompt. In contrast, the other models were fine-tuned using one or a few fixed prompts specifically designed for OCR tasks. So to use Qwen3-VL, we recommend experimenting with prompts.
Here’s a small demo for you to try some of the latest models and compare their outputs.
Evaluating Models
Benchmarks
There’s no single best model, as every problem has different needs. Should tables be rendered in Markdown or HTML? Which elements should we extract? How should we quantify text accuracy and error rates? 👀
While there are many evaluation datasets and tools, many don’t answer these questions. So we suggest using the following benchmarks:
OmniDocBenchmark: This widely used benchmark stands out for its diverse document types: books, magazines, and textbooks. Its evaluation criteria are well designed, accepting tables in both HTML and Markdown formats. A novel matching algorithm evaluates the reading order, and formulas are normalized before evaluation. Most metrics rely on edit distance or tree edit distance (tables). Notably, the annotations used for evaluation are not solely human-generated but are acquired through SoTA VLMs or conventional OCR methods.
OlmOCR-Bench: OlmOCR-Bench takes a different approach: they treat the evaluation as a set of unit tests. For example, table evaluation is done by checking the relation between selected cells of a given table. They use PDFs from public sources, and annotations are done using a wide range of closed-source VLMs. This benchmark is quite successful to evaluate on the English language.
CC-OCR (Multilingual): Compared to the previous benchmarks, CC-OCR is less preferred when picking models, due to lower document quality and diversity. However, it’s the only benchmark that contains evaluation beyond English and Chinese! While the evaluation is far from perfect (images are photos with few words), it’s still the best you can do for multilingual evaluation.
When testing different OCR models, we've found that the performance across different document types, languages, etc., varies a lot. Your domain may not be well represented in existing benchmarks! To make effective use of this new generation of VLM-based OCR models we suggest aiming to collect a dataset of representative examples of your task domain and testing a few different models to compare their performance.
Cost-efficiency
Most OCR models are small, having between 3B and 7B parameters; you can even find models with fewer than 1B parameters, like PaddleOCR-VL. However, the cost also depends on the availability of optimized implementations for specialized inference frameworks. For example, OlmOCR-2 comes with vLLM and SGLang implementations, and the cost per million pages is 178 dollars (assuming on H100 for $2.69/hour). DeepSeek-OCR can process 200k+ pages per day on a single A100 with 40GB VRAM. With napkin math, we see that the cost per million pages is more or less similar to OlmOCR (although it depends on your A100 provider). If your use case remains unaffected, you can also opt for quantized versions of the models. The cost of running open-source models heavily depends on the hourly cost of the instance and the optimizations the model includes, but it’s guaranteed to be cheaper than many closed-source models out there on a larger scale.
Open OCR Datasets
While the past year has seen a surge in open OCR models, this hasn't been matched by as many open training and evaluation datasets. An exception is AllenAI's olmOCR-mix-0225, which has been used to train at least 72 models on the Hub – likely more, since not all models document their training data.
Sharing more datasets could unlock even greater advances in open OCR models. There are several promising approaches for creating these datasets:
Synthetic data generation (e.g., isl_synthetic_ocr)
VLM-generated transcriptions filtered manually or through heuristics
Using existing OCR models to generate training data for new, potentially more efficient models in specific domains
Leveraging existing corrected datasets like the Medical History of British India Dataset, which contains extensively human-corrected OCR for historic documents
It's worth noting that many such datasets exist but remain unused. Making them more readily available as 'training-ready' datasets carries a considerable potential for the open-source community.
Tools to Run Models
We have received many questions about getting started with OCR models, so here are a few ways you can use local inference tools and host remotely with Hugging Face.
Locally
Most cutting-edge models come with vLLM support and transformers implementation. You can get more info about how to serve each from the models’ own cards. For convenience, we show how to infer locally using vLLM here. The code below can differ from model to model, but for most models it looks like the following.
vllm serve nanonets/Nanonets-OCR2-3BAnd then you can query as follows using e.g. OpenAI client.
from openai import OpenAI
import base64
client = OpenAI(base_url="http://localhost:8000/v1")
model = "nanonets/Nanonets-OCR2-3B"
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
def infer(img_base64):
response = client.chat.completions.create(
model=model,
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{img_base64}"},
},
{
"type": "text",
"text": "Extract the text from the above document as if you were reading it naturally.",
},
],
}
],
temperature=0.0,
max_tokens=15000
)
return response.choices[0].message.content
img_base64 = encode_image(your_img_path)
print(infer(img_base64))Transformers
Transformers provides standard model definitions for easy inference and fine-tuning. Models available in transformers come with either official transformers implementation (model definitions within the library) or “remote code” implementations. Latter is defined by the model owners to enable easy loading of models into transformers interface, so you don’t have to go through the model implementation. Below is an example loading Nanonets model using transformers implementation.
# make sure to install flash-attn and transformers
from transformers import AutoProcessor, AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained(
"nanonets/Nanonets-OCR2-3B",
torch_dtype="auto",
device_map="auto",
attn_implementation="flash_attention_2"
)
model.eval()
processor = AutoProcessor.from_pretrained("nanonets/Nanonets-OCR2-3B")
def infer(image_url, model, processor, max_new_tokens=4096):
prompt = """Extract the text from the above document as if you were reading it naturally. Return the tables in html format. Return the equations in LaTeX representation. If there is an image in the document and image caption is not present, add a small description of the image inside the <img></img> tag; otherwise, add the image caption inside <img></img>. Watermarks should be wrapped in brackets. Ex: <watermark>OFFICIAL COPY</watermark>. Page numbers should be wrapped in brackets. Ex: <page_number>14</page_number> or <page_number>9/22</page_number>. Prefer using ☐ and ☑ for check boxes."""
image = Image.open(image_path)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": [
{"type": "image", "image": image_url},
{"type": "text", "text": prompt},
]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
return output_text[0]
result = infer(image_path, model, processor, max_new_tokens=15000)
print(result)MLX
MLX is an open-source machine learning framework for Apple Silicon. MLX-VLM is built on top of MLX to serve vision language models easily. You can explore all the OCR models available in MLX format here. They also come in quantized versions.
You can install MLX-VLM as follows.
pip install -U mlx-vlmwget https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/throughput_smolvlm.png
python -m mlx_vlm.generate --model ibm-granite/granite-docling-258M-mlx --max-tokens 4096 --temperature 0.0 --prompt "Convert this chart to JSON." --image throughput_smolvlm.png Remotely
Inference Endpoints for Managed Deployment
You can deploy OCR models compatible with vLLM or SGLang on Hugging Face Inference Endpoints, either from a model repository “Deploy” option or directly through Inference Endpoints interface. Inference Endpoints serve the cutting-edge models in a fully managed environment with GPU acceleration, auto-scaling, and monitoring without manually managing the infrastructure.
Here is a simple method of deploying nanonets using vLLM as the inference engine.
Navigate to the model repository
nanonets/Nanonets-OCR2-3BClick on the “Deploy” button and select the “HF Inference Endpoints”
Configure the deployment setup within seconds
After the endpoint is created, you can consume it using the OpenAI client snippet we provided in the previous section.
You can learn more about it here.
Hugging Face Jobs for Batch Inference
For many OCR applications, you want to do efficient batch inference, i.e., running a model across thousands of images as cheaply and efficiently as possible. A good approach is to use vLLM's offline inference mode. As discussed above, many recent VLM-based OCR models are supported by vLLM, which efficiently batches images and generates OCR outputs at scale.
To make this even easier, we've created uv-scripts/ocr, a collection of ready-to-run OCR scripts that work with Hugging Face Jobs. These scripts let you run OCR on any dataset without needing your own GPU. Simply point the script at your input dataset, and it will:
Process all images in a dataset column using many different open OCR models
Add OCR results as a new markdown column to the dataset
Push the updated dataset with OCR results to the Hub
For example, to run OCR on 100 images:
hf jobs uv run --flavor l4x1 \
https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nanonets-ocr.py \
your-input-dataset your-output-dataset \
--max-samples 100The scripts handle all the vLLM configuration and batching automatically, making batch OCR accessible without infrastructure setup.
Going Beyond OCR
If you are interested in document AI, not just OCR, here are some of our recommendations.
Visual Document Retrievers
Visual document retrieval is to retrieve the most relevant top-k documents when given a text query. If you have previously worked with retriever models, the difference is that you search directly on a stack of PDFs. Aside from using them standalone, you can also build multimodal RAG pipelines by combining them with a vision language model (find how to do so here). You can find all of them on Hugging Face Hub.
There are two types of visual document retrievers, single-vector and multi-vector models. Single-vector models are more memory efficient and less performant; meanwhile, multi-vector models are more memory hungry and more performant. Most of these models often come with vLLM and transformers integrations, so you can index documents using them and then do a search easily using a vector DB.
Using Vision Language Models for Document Question Answering
If you have a task at hand that only requires answering questions based on documents, you can use some of the vision language models that had document tasks in their training tasks. We’ve observed users trying to convert documents into text and passing the output to LLMs, but if your document has a complex layout, and your converted document outputs charts and so on in HTML, or images are captioned incorrectly, the LLM will miss out. Instead, feed your document and query to one of the advanced vision language models like Qwen3-VL not to miss out on any context.
Wrapping up
In this blog post, we wanted to give you an overview of how to pick your OCR model, existing cutting-edge models and capabilities, and the tools to get you started with OCR.
If you want to learn more about OCR and vision language models, we encourage you to read the resources below.
readers loved this