TL;DR:
An assistant guesses which window you mean from clues: which one has focus, which you used last, and what your words refer to.
When the clues disagree, the right move is a short question, not a confident guess.
You can help by naming the window or clicking it before you ask.
You have a recipe in one window and a shopping list in another. You say "double this" and the assistant starts doubling the shopping list. Nothing is broken: you said "this", and the assistant had to decide what "this" was. People do that effortlessly because we share a room with the person we are talking to. A computer has to work it out from clues.
The clues an assistant can use
Three clues carry most of the weight. The first is focus: the window that is active and receives your typing. The second is recency: which window you were in a moment ago, and which has changed most lately. The third is the words you used: "the recipe" or "this table" suggests something about what you mean.
Other clues exist too. The window under your cursor, a visible title, an obvious match between your question and a page's content. None is certain alone, so a careful assistant combines them.
What happens step by step
You ask a question that uses a word like "this" or "that".
The assistant looks at which window has focus and what was just on screen.
It compares your words with the content of the candidate windows.
If one window clearly fits, it proceeds and says which window it used.
If two fit about equally, it asks you to pick.
Walk-through: asking about "this" with two windows open
You have two windows open side by side: a recipe on the left that serves 4, and a shopping list on the right. Your cursor last clicked in the list, so the list has focus. You say, "Double this."
Focus points to the shopping list, because that is where you clicked last.
The words say "double", which fits quantities. The recipe has quantities. The list has item names with few numbers.
Recency is weak: you read the recipe for several minutes, then clicked the list once.
The clues disagree: focus says list, words and content say recipe. A well-behaved assistant stops here.
It asks: "Do you mean the recipe on the left? It serves 4." You say yes, and it doubles the quantities: 4 servings becomes 8, and each amount is multiplied by 2.
One question took two seconds. The wrong guess would have cost you a rewritten list, and your trust in the tool. Check the result against the page: a recipe line of 250 grams becomes 500 grams, and 2 eggs becomes 4.
How you can help it
Click the window you mean before you ask.
Name it: "the recipe", "the spreadsheet", "the left window".
Close or minimize windows you do not need during the question.
Ask about one thing at a time.
These habits also help human helpers. Pointing at something removes the guesswork.
Why guessing confidently is the wrong default
A wrong answer given confidently looks the same as a right one. When the cost of being wrong is high, such as sending, deleting or changing something, the helper should confirm which window it means before acting. When the cost is low, such as answering a question, it can pick the likeliest window and say which one it used, so you can correct it quickly.
What it cannot do
It cannot see windows hidden behind others or minimized, unless they are captured.
It cannot always tell two similar windows apart, such as two spreadsheets.
It cannot read your intent from nothing. A vague "this" with several equal candidates always needs a question.
Privacy and control
Several windows on screen means several things the assistant might see: a private message behind a work document, for instance. Hide or close what you do not want considered before you ask. Choose a helper that looks only while you invoke it, and that tells you which window it used.
A second example: two similar windows
Suppose you have two browser windows open, both showing a document, and you say "summarize this." Focus gives one answer. If the titles differ, the assistant can read both and notice that only one matches your question. If the titles are the same, no clue separates them, and the only honest move is to ask which one.
This is also why saying a few extra words is cheap insurance. "Summarize this one, the one with the budget" gives the assistant a content clue to match against, and turns a coin toss into a certainty.
Try it yourself
DuckyHelper is one tool that works this way: press one key in any app (fn on a Mac, Ctrl+Shift+Space on Windows and Linux), ask out loud, and it answers out loud while you keep working. Because it sees your screen only when you ask, you can click the window you mean and say "explain this" while the pill is open, and it answers out loud about that window. You can try it for 10 minutes with no card.
Frequently asked questions
Why did it pick the wrong window?
The clues pointed in different directions. Click the window you mean or name it, and ask again.
Should it always ask first?
No. Constant questions are tiresome. It should ask when the clues conflict or when the action could not easily be undone.
Get started
