TL;DR:
An assistant that sees your screen takes a picture of it when you ask, reads the text and layout in that picture, and answers about what is there.
It sees a snapshot, not your whole computer: it cannot see what is hidden, off screen or covered by another window.
A well-built one captures only when you ask, and tells you so.
You are halfway through a form and a dialog box pops up with a message you do not understand. You could retype it into a search box, or describe it to a friend and hope you got the wording right. An assistant that can see your screen skips that step: you say "what does this mean?" and it already knows which dialog you mean.
The short answer
Seeing a screen is less mysterious than it sounds. The assistant takes a screenshot, which is just an image of what your display is showing at that moment. Then software turns the image into something it can reason about: the words that appear, where they sit, and what kind of thing each part looks like (a button, a menu, a chart, a paragraph).
After that, answering is the same as answering any question, except the assistant has the picture next to your words. "What does this mean?" stops being vague, because "this" is on the screenshot.
What happens step by step
You ask. You press a key and speak. Nothing needs to be captured until this moment.
The screen is captured. The computer grabs the pixels the display is showing. This is the same thing that happens when you press the screenshot shortcut yourself.
The picture is read. Text is recognised, and the layout is worked out: what is a title, what is a button, what is body text. Reading text in images has its own article, linked below.
Your question is matched to the picture. The words "this" and "that" are connected to what is visible, usually the window in front.
An answer is made and delivered. It can be spoken, shown as text, or drawn on top of your screen so you can see what it means.
For more on step 3, see reading text on a screen: OCR explained. For how an assistant picks one button out of dozens, see how an AI finds the right button on a screen.
Walk-through: asking about an error dialog
Say you are saving a file and a box appears. Its title is "Cannot save" and the message says the file is in use by another program. There are two buttons, "Retry" and "Cancel".
You press the key and say, "Why can't I save this?" The assistant captures the screen and reads the dialog: a title, one sentence of explanation, two buttons. It also notices that the dialog sits on top of a spreadsheet window, with the file name in that window's title bar.
Its answer combines both: the file is probably open somewhere else, maybe in another window or another program, so close it there and press Retry. If you ask "which button do I press?", it can answer "Retry, after you close the other copy", and with a drawing on screen it can circle that button. Notice what made this possible. It did not guess from your description. It read the actual words in front of you, including the file name you might not have thought to mention.
What it cannot do
See what is not displayed. Windows that are minimised, behind other windows or scrolled out of view are not in the screenshot. If the answer is further down the page, scroll first.
Read what is too small or too faint. Tiny grey text on a grey background can be missed or misread. Zooming in on the app helps.
Know what you meant by "this" when the screen is busy. With five windows tiled side by side, say which one: "the dialog in the middle".
Be sure about unusual interfaces. A custom-drawn app with unlabeled icons is harder than a standard form. A good assistant says it is unsure instead of pretending.
Snapshot, not a recording: A screenshot is a snapshot. It does not include sound, motion between frames, or anything that happened before you asked.
Privacy and control
Whether it is safe to let an assistant see your screen depends on one thing: when does it look? An assistant that captures only when you ask is very different from one that captures all day.
A good design looks only while the pill is open and for the answer to that question, and does nothing when idle. Before you use any assistant that sees your screen, check three things: when it captures, where the capture goes, and whether you can see and control that. If the product does not say clearly, treat that as an answer.
One practical habit helps with any such tool: close or hide windows with private information (banking, messages, medical portals) before you ask a question about something else.
Try it yourself
You can see the idea without any assistant. Take a screenshot of an error message, then read only the screenshot as if you were seeing it for the first time. Everything you can work out from that image is everything a screen-seeing assistant has to work with.
If you want to try the real thing, DuckyHelper is on Mac, Windows and Linux. Press fn on a Mac, or Ctrl+Shift+Space on Windows and Linux, and ask about whatever is on your screen. You can try it for 10 minutes, no card.
Frequently asked questions
Does the assistant see my whole computer?
It sees what your display shows when it captures: the visible windows. It does not see files on disk, hidden windows or other devices unless they are on screen.
Is a screen-seeing assistant the same as screen recording?
No. Recording keeps a continuous video. Answering a question needs a single snapshot taken when you ask.
Why not just copy and paste the error?
You can, but you lose the layout, the buttons and the surrounding window, and some messages cannot be selected at all. The screenshot keeps all of it.
Get started
