TL;DR:
Pointing removes ambiguity: your cursor tells the assistant which word, button or picture "this" refers to.
The assistant uses the pixels around the cursor, reads them, and answers about exactly that.
Ask a short, specific question, and say so if the answer is about the wrong thing.
You are reading an article and hit a word you do not know: "amortization". You could select it, copy it, open another tab, paste it and read a definition. Or you could just point at it and ask "what's this?" The second way keeps your eyes on the page. It only works if the assistant knows which word you pointed at.
Why pointing beats describing
Describing something on a screen in words is clumsy. "The third word in the second paragraph, in bold" takes longer than the point itself. Pointing is also how people have always handled "this" and "that". A cursor is the closest thing a computer has to a finger.
What happens step by step
You move your cursor to rest on what you are curious about.
You ask out loud or in text, using "this" or "that".
The assistant captures the screen and notes where the cursor is.
It looks at the area around that position and reads the text or picture there.
It answers about that item, and may show the answer next to the cursor.
The cursor position is a coordinate on the screen. If it is converted correctly, the assistant looks at the right spot.
Walk-through: pointing at a word with your cursor
Say you are reading a loan article, and your cursor rests on the word "amortization" in the middle of a sentence.
You ask, "What does this mean?"
The assistant takes a screenshot and finds the cursor at, say, a position in the second paragraph.
It reads the characters near that point and picks the word the cursor sits on. If the cursor is between two words, it reads the closer one, and a short check such as "You mean amortization?" settles it.
It reads the surrounding sentence, since a word's meaning depends on context.
It answers: amortization is spreading a loan's repayment over regular payments so that it is fully paid by the end.
The answer in that last step is a standard definition. What the assistant adds is not knowing more words than a dictionary, but knowing which word you meant and in which sentence.
When there is no pointer to go on
Some assistants look at the whole screen when you ask, not at a spot under the cursor. Then "this" has to come from your words and from what is in view. A short detail does the job: "the bold word in the second paragraph", "the tallest bar", "the total at the bottom". A good assistant also shows you which part it took you to mean, by drawing a circle or an arrow on it, so a wrong guess is easy to spot. If it marks the wrong thing, say which one you meant and ask again.
Getting good answers from a point
Keep the cursor still on the target for a moment before you ask.
Point at the word itself, not at white space next to it.
Ask a specific question: "what does this mean?" or "why is this red?"
If the answer is about the wrong thing, say so. Moving the cursor and asking again takes seconds.
What it cannot do
It cannot know what you meant if your cursor sits on empty space.
Tiny text or crowded screens can make the target ambiguous.
It cannot read text that is not visible at all, such as an image of text at a very small size.
It can be wrong about context. Check anything important.
Privacy and control
A cursor is part of what you are doing, and a tool that tracked it all day would know a lot about you. Prefer a helper that only looks when you invoke it, and not before. The pointed-at word is all it needs for your question.
Pointing at things that are not words
The same idea works for a chart, an icon or a photo. Rest the cursor on a bar in a chart and ask "why is this one so tall?" The assistant reads the label under the bar, the axis, and the neighbouring bars, then answers about that bar. For an unfamiliar icon in a toolbar, it can name it and say what it usually does, though you should confirm in the app's own help before clicking something that changes data.
Pointing is strongest when the target is small and the question is short. For a big, complex region, such as a whole page, describing what you want in words works better than pointing.
Try it yourself
With DuckyHelper you press one key in any app (fn on a Mac, Ctrl+Shift+Space on Windows and Linux), ask out loud about what is on your screen, and it answers out loud while you keep working. It sees your screen and draws on it, so you can ask about a word or a picture right where you are reading. You can try it for 10 minutes with no card.
Frequently asked questions
Can an assistant point back at me?
Yes. Many can draw a mark or a circle on your screen to show you the part they mean, which is often clearer than words.
What if two things overlap under the cursor?
A good assistant picks the visible top item and, if unsure, asks which one you mean.
Get started
