TL;DR:
Drawing on your screen uses a transparent window that sits above everything and ignores mouse clicks.
To land on the right spot, the shape needs screen coordinates, adjusted for display scaling and for where each monitor sits.
The marks should be temporary and clear out of the way, so they help without leaving anything behind.
A friend sitting next to you would just point: "That cell, the one with the red corner." An assistant that can only speak has to describe it instead: "the third column, fourteenth row, the one containing the formula that starts with equals." Pointing is faster and nearly impossible to misunderstand. So how does software point at something on a screen it does not own?
The short answer
The trick is an overlay: a window with no background, no border, and nothing in it except the shapes to be shown, placed above all other windows. You see through it to the app below, with the circle or arrow floating on top.
Two further details make it useful. The overlay must be click-through, so that your mouse presses pass straight to the app underneath. And the shapes must be placed using real screen coordinates, so a circle lands on the right cell and not near it.
What happens step by step
Find the target. From the screenshot, the assistant locates the thing to point at and gets its position in image pixels.
Convert to screen coordinates. Adjust for the difference between the screenshot's size and the real display's size.
Create the overlay. A transparent window the size of the display, flagged as click-through and always on top.
Draw. Add the circle, arrow or underline at the computed position, perhaps animated so your eye catches it.
Remove it. After a few moments, or when you click or scroll, take the marks away.
Walk-through: an assistant circling the cell with a wrong formula
Say your display is 1920 pixels wide and the assistant works from a screenshot scaled down to 960 pixels wide. It finds the faulty cell at x = 480, y = 300 in the screenshot. The scale factor is 1920 / 960 = 2, so on the real screen the cell is at (480 × 2, 300 × 2) = (960, 600).
Skipping that step would put the circle at (480, 300), halfway to where it should be, and the circle would sit on a completely different cell. Checking the result is simple: the real position divided by the scale factor gives back the screenshot position, 960 / 2 = 480 and 600 / 2 = 300, so the numbers are consistent.
With the correct position, the overlay draws a ring around (960, 600) and the assistant says, "This cell: it adds the total of the wrong column." Because the overlay ignores clicks, you can click into the cell right through the ring to fix it.
Scaling and multiple monitors
Two things complicate coordinates in practice. Display scaling: sharp displays often draw each point of the interface using several physical pixels, so the numbers an app uses and the numbers in a screenshot may differ by a factor of two or more. Multiple monitors: each has its own position in one shared coordinate space, and they may have different scaling from each other.
Case | What to account for |
|---|---|
Sharp display | Scale between screenshot pixels and interface points |
Two monitors side by side | Add the monitor's offset so the point lands on the right screen |
Monitors with different scaling | Use each monitor's own factor, not one for all |
Window moved after the screenshot | Re-measure, because the target has moved |
Why the marks should be temporary
A mark that stays on screen is a liability. It covers content, it points at things that have since moved, and it clutters your work. So a well-designed overlay shows marks only as long as they help, and clears them when you click, scroll or move on.
Think of it as a finger on a page rather than ink. The finger shows the spot, and when you look away it is gone. That keeps the overlay honest: whatever you see circled is circled now, about this screen as it is at this moment.
What it cannot do
Draw on things that move. If the page scrolls after the screenshot, the mark is where the cell used to be. This is why marks should vanish on scroll or click.
Draw over some protected content such as certain full-screen video or system prompts, depending on the operating system.
Be pixel perfect on a blurry guess. If the assistant misjudged the target, a precise circle around the wrong thing is still wrong.
Privacy and control
An overlay only displays marks. It does not read other apps by itself. Reading comes from the screenshot step, which should happen only when you ask. DuckyHelper draws on your screen and looks only while the pill is open and for that answer.
Try it yourself
Take a screenshot, open it in an image viewer, note the pixel size of the picture, then compare it to your display's resolution in system settings. If they differ, you have just found the scale factor an overlay has to correct for.
To see drawing in action, download DuckyHelper and ask about something on screen. You can try it for 10 minutes, no card.
Frequently asked questions
Does the overlay block my clicks?
It should not. A well-built overlay is click-through, so your clicks go to the app below.
Why do the marks disappear?
Because they would otherwise point at the wrong place once the screen changes. Clearing on click or scroll prevents that.
Can it draw on a second monitor?
Yes, if it adds that monitor's offset and uses its scaling.
Get started
