TL;DR:
To find a button, an assistant combines three clues: the words on or near it, the shape of its icon, and where it sits in the layout.
When two candidates are close, a careful assistant asks or points instead of guessing.
Screens it has never seen are harder, and unlabeled icons are the hardest case.
You open an app you have never used and need to save your work. The toolbar has twenty small icons and no word "Save" anywhere. You scan, you hesitate, you hover over a few. An assistant that can see the screen faces the same puzzle, and it solves it with the same clues you use.
The short answer
Every button gives away what it is in three ways. It may carry a label, the word printed on it. It has a look, such as a floppy disk, a gear or a magnifying glass. And it has a position, because interfaces follow habits: close is in a corner, search is near the top, the main action sits at the end of a dialog.
An assistant weighs all three together. A word is the strongest clue. An icon is a good clue if it is a common one. Position settles ties and confirms a guess.
What happens step by step
List the candidates. From the screenshot, work out which regions are clickable things: boxes with text, icons, menu entries.
Read each one. Collect the label if there is one, or describe the icon if there is not.
Score them against the request. "Save" matches the word Save, a disk icon, and the words "Save as" in a menu.
Check the neighbours. A button inside a dialog titled "Delete account" means something different from the same word in a toolbar.
Decide, or ask. If one candidate is clearly best, name or point to it. If two are close, say so.
Walk-through: finding the Save button in an unfamiliar app
Imagine a drawing app. The top bar holds, left to right: a folder icon, a disk icon, an arrow curving left, an arrow curving right, and a gear. You ask, "How do I save this?"
The assistant goes through its clues. No button has the word Save on it. The disk icon is the classic symbol for saving, and it sits right next to the folder icon, which usually means open. That pair, open then save, is a common arrangement, which raises its confidence. The curved arrows are undo and redo, and the gear is settings, so those score low.
It also checks the menus. If the screenshot shows a "File" menu, it knows that menu very probably contains "Save", and that is a second route. Its answer might be: "Click the disk icon, second from the left, or open File and choose Save." Notice it gave a reason and a backup. That is what you want from a helper: not just a location, but the evidence, so you can check it.
When two buttons look alike
Ambiguity is where interfaces get dangerous. A dialog might have "Save" and "Save and close", or two disk icons with different meanings. A careful assistant treats this as a reason to slow down.
Situation | Good behaviour |
|---|---|
One clear label match | Point to it and say why |
Two near matches | Name both and say how they differ |
No label, common icon | Point to it, say it is a guess from the icon |
No label, unknown icon | Say it is unsure and suggest hovering to read the tooltip |
What it cannot do
Know what an icon means in a one-off design. An abstract custom symbol with no label carries no meaning to anyone, person or software.
See buttons that are not drawn yet. Menus that appear on hover or click are not in the screenshot until opened.
Tell a button from a picture of a button. A screenshot embedded in a web page looks clickable and is not.
Privacy and control
Finding a button needs the screenshot, so the same rule applies as for any screen-seeing tool: it should capture when you ask, not constantly.
If you let an assistant click for you, the stakes rise, because a wrong guess is now an action. A careful one asks first before it sends a message or pays for anything. For plain "where is the button" questions, pointing at it and letting you click is always the safest option.
Try it yourself
Take a screenshot of an app you rarely use and cover the labels with your thumb. Try to name five buttons from icon and position alone. You will find you succeed on the common ones (search, close, settings) and fail on the odd ones, which is exactly how an assistant behaves.
To ask about a real screen, download DuckyHelper and press the key while you ask where something is. It looks only while the pill is open and for that answer. You can try it for 10 minutes, no card.
Frequently asked questions
Does it need the buttons to have text?
No, but text helps a lot. Without it, it relies on common icons and position, and should say it is less certain.
Why does it sometimes point to the wrong button?
Usually because two buttons looked alike, or the real one was hidden in a menu that was closed. Asking again with the menu open fixes most cases.
Is this the same as the way screen readers work?
Related but different. Screen readers use the labels apps publish about themselves, covered in the accessibility trees article.
Get started
