Reading text on a screen: OCR explained

TL;DR:

• OCR (optical character recognition) turns an image of text into actual characters you can copy and search.

• It works in stages: find the text, split it into lines and characters, recognise each, then rebuild words and layout.

• Small, low-contrast, stylised or skewed text is where it makes mistakes, so check numbers.

A colleague sends you a screenshot of a table. You need the numbers in your spreadsheet, and the screenshot is just pixels: you cannot select a single digit. Retyping twenty rows is slow and easy to get wrong. This is the job OCR was made for.

The short answer

OCR stands for optical character recognition. It takes an image that contains writing and produces the text itself, as characters the computer understands. To a computer, a photo of the word "cat" and the word "cat" typed in a document are completely different things. OCR bridges them.

Screen-seeing assistants rely on it, because most of what is on a screen reaches them as an image. Once the pixels become text, the assistant can search it, quote it and answer questions about it.

What happens step by step

1. Clean up the image. Convert to a simple contrast, straighten tilted text, and remove noise.

2. Find where the text is. Separate text regions from pictures and empty space.

3. Split into lines, then words, then characters or strokes. Modern systems often read whole lines at once instead of one letter at a time.

4. Recognise. For each piece, choose the most likely character. Context helps: in a column of numbers, a shape that could be "O" or "0" is probably zero.

5. Rebuild. Put characters back into words, lines and, if the software is good, rows and columns.

Walk-through: turning a screenshot of a table into text

Suppose the screenshot shows a small table with the header row "Item, Qty, Price" and three rows below it. Here is what the recogniser outputs when the text is clear:

Item     Qty   Price
Pens     12    1.50
Paper    4     3.20
Tape     6     0.90

Now the part that matters: checking. The easy first test is arithmetic. If the original table has a total line, add the rows. Using the values above, 12 × 1.50 = 18.00, 4 × 3.20 = 12.80 and 6 × 0.90 = 5.40, and 18.00 + 12.80 + 5.40 = 36.20. If the screenshot says the total is 36.20, every digit probably came through. If it says 38.20, one digit was misread somewhere, and now you know to look.

Layout is the other risk. Plain text is read in order, but a table has structure. A good tool keeps columns aligned; a weaker one runs a row into a single line, "Pens 12 1.50", and you must split it yourself. Either way, paste the result into a spreadsheet and look before you trust it.

Where it goes wrong

• Look-alike characters. 0 and O, 1 and l, 5 and S, rn and m.

• Small or blurry text. Fewer pixels per letter means fewer clues. Zoom in before the screenshot.

• Low contrast. Light grey on white, or text over a photo, is hard to separate from the background.

• Unusual fonts and curved or rotated text. Decorative lettering was not what most recognisers learned from.

• Mixed languages and symbols. Maths notation, currency signs and accents are often dropped or swapped.

Check the numbers: Copy numbers from screens with OCR only as a first draft. For money, doses, dates and codes, compare against the source or use a check such as a total.

Privacy and control

OCR reads whatever is in the picture, including things you did not mean to share: a name in a corner, an email in a notification. So the question is the same as for any screenshot: when is it taken, and what does it include?

A tool that is built carefully captures only when you ask, not when idle. If a window has private details, move or close it first. Many apps also let you select and copy text directly; when that works, it beats OCR every time because nothing has to be guessed.

Reading numbers versus reading words

Words are forgiving. If one letter of "table" is wrong, you still know the word, and so does a spell checker. Numbers are not forgiving, because every digit is a valid digit. A 6 read as an 8 produces a perfectly plausible wrong number, and nothing flags it.

That is why the arithmetic check in the walk-through matters more than any quality score. Where a table has no totals, read two or three values back against the original by eye, and pick the ones with the most digits, since those have the most chances to go wrong.

Try it yourself

Take a screenshot of any table, then try your computer's built-in tool for copying text from images if it has one. Compare the output to the picture, line by line, and note which characters it got wrong.

Or download DuckyHelper, press the key and ask it to read a table on your screen back to you. The 10 minute try needs no card.

Frequently asked questions

Is OCR the same as the assistant understanding the text?

No. OCR only produces characters. Understanding what they mean is a separate step that comes after.

Can OCR read handwriting?

Sometimes, but neat print is far easier than cursive. Expect more errors and check the result.

Why not just copy the text from the app?

Copying is better when it works. OCR is for the cases where it does not: images, locked pages, videos and some remote windows.