Scanning Archives for OCR: The Settings That Decide Accuracy
Written for archivists, librarians, genealogists, and anyone about to put a box of documents on a scanner.
When an OCR or handwriting-recognition project disappoints, the conversation usually turns to the model. Which engine, how much training data, whether a custom pipeline would do better. Those are real questions. But in the projects behind the guides on this site, the most common cause of poor accuracy was settled much earlier, at the moment someone chose the scanner settings.
A model can learn a new script. It cannot recover a stroke that the scan never captured. As the Hebrew OCR guide puts it: you cannot un-blur a page later.
This post is about the decisions that happen before any AI touches your collection, and how to check them cheaply.
Resolution: the floor and the standard
The numbers on this site have been consistent for a reason. 300 DPI is the floor for OCR. 400–600 DPI is the standard for historical material.
DPI on its own is a proxy, though. What the recognition model actually needs is enough pixels across each letter to tell similar shapes apart. Hebrew resh and dalet, vav and zayin, Cyrillic letters that were abolished in 1918 and differ from their modern look-alikes by a single fine stroke. So the right resolution depends on the size of the writing, not on the size of the page:
- Clear print at book size. 400 DPI is comfortable; 300 is workable.
- Dense newsprint, small type, footnotes. Go to 600. Small characters are where low resolution hurts first.
- Handwriting, marginalia, faded ink. 400 at minimum, 600 when the hand is small or the ink is weak.
It is tempting to save storage by scanning lower. Do the arithmetic before you decide. An A4 page at 400 DPI in uncompressed 24-bit colour is about 46 MB; at 300 DPI it is about 26 MB. Lossless compression reduces both. Against the cost of re-handling a fragile collection a second time, the difference is small.
Colour, greyscale, and the one setting to avoid
Capture masters in colour where you can, or at least in 8-bit greyscale.
Never capture historical material in bitonal (1-bit black-and-white) mode. Bitonal scanning makes a hard decision at every pixel: ink or paper. On a clean modern page that decision is easy. On a foxed, stained, or faded page, the scanner's threshold throws away exactly the faint strokes and fine distinctions that recognition depends on, and nothing downstream can bring them back.
Colour also carries information beyond aesthetics. Different inks, pencil annotations, stamps, and later additions are often easier to separate in colour, and a cataloguer will want that distinction even if the OCR engine does not.
File format: keep a clean master
Treat the scan as two products.
- The master. Uncompressed or losslessly compressed TIFF (lossless JPEG 2000 is also common). This is the file you preserve and the file you run recognition on.
- Derivatives. Compressed JPEGs or PDFs for viewing, sharing, and the web.
Heavy JPEG compression creates artefacts precisely around the edges of strokes. That is invisible to a reader at normal zoom and very visible to a model.
Do not bake enhancements into the master. Sharpening, contrast boosts, automatic "document mode" cleaning, and aggressive auto-cropping all feel helpful and all make irreversible decisions. If a page needs enhancement for recognition, do it in the pipeline, on a copy, where it can be tuned and undone.
Margins, versos, and the edges of the page
Automatic cropping is a quiet source of data loss. Keep a visible margin around the whole leaf. Archival documents put important information at the edges:
- folio and page numbers,
- marginal notes and corrections,
- stamps, registration numbers, and archive signatures,
- the edge of a table that continues on the facing page.
Scan versos even when they look blank. A faint pencil note on the back of a photograph or a form is often the most useful line in the file, as the self-configuring cataloguing post describes for verso handwriting.
Light, flatness, and bound volumes
Even, diffuse light matters more than an expensive sensor. Glare on glossy paper and shadows from a raking lamp both read as noise.
Bound volumes add the problem of the gutter. Text curves into the binding, lines bend, and the inner column shrinks and darkens. Recognition engines handle a little curvature, but not much. Where the object allows it, a book cradle that supports the volume at an opening angle helps. A glass plate flattens the page, but only use one when a conservator agrees that the binding and paper can take it.
For genealogists working with a phone rather than a scanner, the same principles apply at smaller scale: lay the page flat, use daylight rather than the flash, hold the camera parallel to the page, and fill the frame while leaving a margin.
Reference targets and published guidelines
You do not need to invent a quality standard. Two widely used ones exist:
- FADGI (Federal Agencies Digital Guidelines Initiative, United States).
- Metamorfoze (the Dutch national programme for preserving paper heritage).
Both define quality levels for different kinds of material and rely on reference targets, charts with known colours, greys, and fine detail, photographed alongside or before the originals so that capture quality can be measured rather than judged by eye. Even if you do not aim for their strictest tier, shooting a target at the start of each session catches drifting settings early.
Order and names
Recognition output is only useful if you can put it back where it came from. Decide the file-naming scheme before the first scan: collection, box, file, and a zero-padded sequence number. Keep the physical order, including blank pages and inserts. A missing page in a numbered series is a cataloguing problem; a silently re-ordered one is worse.
Test on 50 pages before you scan 50,000
The cheapest step in any digitisation project is a pilot.
- Pick 20–50 pages that represent the hard cases. The smallest type, the faintest ink, the tightest gutter, the worst paper.
- Scan them at two or three settings. For example 300 and 400 DPI, greyscale and colour.
- Run recognition on each set and correct a sample by hand.
- Compare error rates. The confidence-calibration case study shows why you should also look at where the errors cluster, not just the average.
If the higher setting reduces errors meaningfully on your hard pages, the storage cost has paid for itself. If it does not, you have evidence for the cheaper choice.
What if the collection is already scanned?
Most collections are. Do not rescan everything on principle.
Run recognition on the existing scans, measure, and find where errors concentrate. Usually it is a minority of pages: the bitonal batch from an old project, the microfilm-derived images, the volumes with a tight binding. For those, and only where the originals survive, a fresh scan from the original usually buys more than any amount of model tuning. A scan made from microfilm is at least one generation away from the paper, and every generation loses detail.
A short checklist
- Resolution: 400 DPI for most historical material, 600 for small type and weak handwriting.
- Colour or 8-bit greyscale. Never bitonal.
- TIFF or other lossless master; compressed derivatives only for viewing.
- No sharpening, cleaning, or tight auto-crop on the master.
- Full margins; scan versos.
- Even light; support bound volumes; ask a conservator before using glass.
- A reference target per session.
- File names and page order fixed before the first scan.
- A pilot of 20–50 hard pages before the full run.
If you are planning a digitisation project and want recognition to work on the result, the time to talk about the model is before the scanner is switched on. See how we approach historical Hebrew OCR and HTR, or get in touch with a sample of your hardest pages.