Searching OCR Text in Historical Archives: What Works
Written for historians, genealogists, archivists and anyone who has typed a name into a digitised newspaper portal and got nothing back.
A search box over a digitised collection looks like a promise: every word on every page, one query away. In practice, when you search OCR'd historical text you are not searching the pages. You are searching a machine's reading of the pages, and that reading has predictable gaps.
"No results" in an OCR'd collection is not evidence that the word is absent. This post explains why, and what to do instead.
Why exact keyword search fails
OCR noise
Even good OCR on historical print makes errors, and they cluster where search needs precision.
- Misread characters. Hebrew resh and dalet, vav and zayin, he and het. In pre-reform Russian, yat (ѣ) read as ь, ъ or б. One wrong letter and an exact query misses the word.
- Broken and merged words. Narrow newspaper columns, faint type and tight spacing split words in two or glue neighbours together.
- Hyphenation at line ends. A word divided across two lines is stored as two fragments unless the pipeline rejoins them.
- Running headers and page furniture. The newspaper title, date and page number repeat on every page. They inflate counts and pollute results for common words.
The arithmetic is unforgiving. The pre-revolutionary Russian OCR post reports roughly 90% character accuracy for a fine-tuned model on 1917 newsprint. If errors fell independently, a ten-letter word would come through intact only about a third of the time (0.9 to the tenth power is about 0.35). Real errors are not independent, but the direction is clear: long words and rare names are exactly the ones exact search loses.
Historical spelling
The text on the page often does not use the spelling a modern user types.
Russian before the 1917–1918 spelling reform used ѣ, і, ѳ and ѵ and a final hard sign ъ after consonants. A modern query for мир will not match миръ or міръ. The fix at the archive level is to keep two layers, diplomatic and normalised, as the Russian OCR post describes. Many portals publish only one.
Hebrew has its own version of the problem. Spelling with and without matres lectionis (ktiv male and ktiv haser) varies by period, publication and writer: שולחן and שלחן, תוכנית and תכנית. Yiddish adds competing orthographies: Soviet Yiddish spelled words of Hebrew origin phonetically, so the title of the Soviet paper Der Emes appears as אמת in its first run (1918–1919) and as עמעס from 1920 on.
Morphology
Hebrew attaches prepositions and conjunctions directly to the word. The letters ו, ה, ב, כ, ל, מ, ש can all appear as prefixes, often stacked. A search for ירושלים misses בירושלים, מירושלים and ושבירושלים unless the engine handles prefixes.
Stripping prefixes automatically has a cost. ברק can be a name or ב + רק. מלך can be "king" or מ + לך. Expect false positives and check them.
Russian and Yiddish inflect heavily. Двоевластие ("dual power") appears as двоевластия, двоевластию, двоевластием, and in pre-reform text as двоевластіе. A prefix query such as двоевласт* catches all of them. Most full-text engines support trailing wildcards; use them by default for inflected languages.
What works instead
1. Build a small variant list per key term
Before searching, write down the forms you expect: old and modern spelling, full and defective spelling, common OCR confusions, transliterations of names, abbreviations. Ten to twenty variants per key term is usually enough to start.
This is the same logic as the domain lexicon for OCR, applied at query time instead of recognition time. Keep the list in a file, add to it every time a concordance shows you a form you missed, and reuse it across projects.
2. Use fuzzy matching carefully
Fuzzy (approximate) search matches words within a small edit distance, typically one or two character changes. Lucene-based engines such as Solr and Elasticsearch express this as term~1 or term~2.
It finds misread forms you could not predict. It also returns many words that are simply similar. The shorter the word, the worse the trade-off. A two-edit fuzzy query on a four-letter word matches a large share of the vocabulary. Reserve fuzzy search for longer words and names, and combine it with a date range or collection filter.
3. Read every hit in KWIC
A KWIC (keyword-in-context) concordance shows each hit on its own line, with the matched word centred and a fixed window of text on either side. Sorted by the words to the right or left, it groups similar uses together.
For OCR'd material it is the researcher's main verification tool:
- false positives from fuzzy or prefix matching are visible at a glance;
- new variant spellings show up in the matched column, ready for your list;
- repeated header text and duplicated pages stand out.
Then go back to the page image. Anything you quote, count or build an argument on must be checked against the scan, not the OCR line.
4. Add semantic search as a complement
Embedding-based search finds passages by meaning, so it can surface a discussion of a topic that never uses your keyword, and it tolerates some OCR noise. RAG for archives builds on this to answer questions with cited passages.
It is a complement, not a replacement. Semantic retrieval returns the most similar passages, not every matching one, so it is weak for exhaustive lists, counts and exact names. Use keyword and KWIC for "every mention of X", semantic search for "where is Y discussed", and verify both against the images.
5. Measure recall before trusting "no results"
Take a set of mentions you already know about: 20 or 30 occurrences you found by reading a sample of pages. Run your queries and count how many are found. If the variant list, wildcards and fuzzy matching together find most of them, a negative result elsewhere starts to mean something. If they find half, a negative result means very little.
The confidence-calibration case study makes the same point about recognition: check where the errors cluster, not only the average.
A checklist for researchers
- Treat "no results" as "not found by this query", never as "not there".
- Write a variant list per key term: spelling, orthography, OCR confusions, transliterations.
- Use prefix wildcards for Hebrew, Russian and Yiddish.
- Use fuzzy matching on long words and names only.
- Read results in KWIC; add new variants to your list.
- Check every quoted or counted hit against the page image.
- Test recall on a known set of mentions before drawing conclusions.
A checklist for archives publishing OCR text
- Expose the raw OCR text layer, ideally downloadable, alongside the images.
- Allow wildcard and fuzzy search, and document the syntax.
- Show confidence. Formats such as ALTO XML can store per-word confidence; surface it, or at least flag low-confidence pages.
- Link every hit to the image, with the matched region highlighted.
- Publish both diplomatic and normalised text where orthography changed.
- State which engine produced the text and when, so users know what they are searching.
The Hebrew OCR engine comparison covers the recognition side; the checklist above is about making that text usable once it exists.
If your collection is already scanned but search results are poor, the problem is usually a mix of text quality and search design, and both can be measured on a sample. See how we approach historical Hebrew OCR and HTR, or get in touch with a few pages and the terms you have been trying to find.
Frequently Asked Questions
Why does full-text search miss words I can see on the page image?
Because the search runs on the OCR text layer, not on the image. If the engine misread one letter, split the word at a line-end hyphen, or the page uses an older spelling or an inflected form, an exact keyword query will not match, even though a human reader sees the word clearly.
Is fuzzy search the solution to OCR errors?
It helps, but it trades precision for recall. Allowing one or two character differences finds misread forms, and on short words it also floods the results with unrelated terms. Use it on longer words and names, and read the hits in context.
What is a KWIC concordance and why should researchers use one?
KWIC (keyword in context) lists every hit as one line, with the matched word centred and a few words on either side. It lets you scan hundreds of hits quickly, discard false matches, and spot variant forms you had not thought to search for.
Can semantic search or RAG replace keyword search in archives?
No. Semantic search is good at finding passages about a topic in different words, but it is unreliable for exact names, exhaustive lists and counts. Use it alongside keyword search, and verify every passage against the page image.