Zoom, scaling and dark mode: why assistants must handle them

TL;DR:

• Scaling means the screen has more physical pixels than the interface's own coordinates. A tool must divide or multiply to point correctly.

• Zoom and dark mode change how things look, so a tool tested only on a default screen can misread or misplace.

• Testing on several scales, zoom levels and themes is the only way to be sure.

Many people make their screen bigger, because text is easier to read that way. Others switch to dark mode at night. These are ordinary settings, and they quietly break any tool that assumes every screen looks like the one it was built on. For an assistant that points at things, these settings matter more than most.

Three settings, three kinds of change

• Display scaling (such as 150%) makes everything larger by mapping more physical pixels to each interface unit.

• Zoom in a browser or an app makes just that content larger, inside its own window.

• Dark mode changes colors, so text and backgrounds swap brightness, and contrast changes.

Scaling and zoom change geometry, so they threaten pointing. Dark mode changes appearance, so it threatens reading. A good assistant has to handle both.

What happens step by step

1. The assistant captures a screenshot. The image is measured in physical pixels.

2. It finds the thing you asked about inside that image.

3. It converts that position into the interface's coordinates, which is what the system uses for drawing and pointing.

4. It draws the mark there.

5. You see the mark on the right thing, whatever your settings.

Step three is the one scaling affects. If a tool skips it, the mark appears too far down and to the right.

Walk-through: a scaled screen shown as a ratio

Suppose a screen has 1920 × 1080 physical pixels and is set to 150% scaling. The interface now behaves as though the screen were smaller, so every unit it uses stands for 1.5 physical pixels.

1. Interface width: 1920 ÷ 1.5 = 1280. Interface height: 1080 ÷ 1.5 = 720. So the interface works on a 1280 × 720 grid.

2. A button sits at physical pixel (960, 540) in the screenshot.

3. Convert to interface coordinates: 960 ÷ 1.5 = 640 and 540 ÷ 1.5 = 360, so (640, 360).

4. Check by going back: 640 × 1.5 = 960 and 360 × 1.5 = 540. It matches.

The ratio is 1.5 physical pixels per interface unit. If the tool used (960, 540) as interface coordinates, it would draw at the physical position, 1.5 times too far across and down. Here that lands in the bottom right instead of the middle, since 960 is far beyond the 640 center of a 1280 grid.

Why zoom and dark mode matter too

Browser zoom changes the size of the page inside the window, not the screen. A tool that finds a button by its remembered shape and size will miss it when the page is zoomed. Matching should rely on what is actually visible in this screenshot, not on a stored picture of the default look.

Dark mode changes contrast. Thin gray text on a dark background can be harder for a tool to read, and for a person. Reading text accurately in both themes is part of doing the job well, along with a high-contrast mode that some people need.

Testing

The only dependable answer is to test. That means trying a range of scales, several zoom levels, both light and dark themes, and more than one display, then checking that a mark lands on the intended thing each time. A tool that has not been tried on your setup may still work, but you should check its first few answers.

What it cannot do

• It cannot always read very small or low-contrast text, especially at high zoom-out.

• It can be thrown by unusual themes, custom fonts or overlapping windows.

• It cannot fix an interface that is hard to see for everyone. Turn on larger text or a higher contrast if you need it.

Privacy and control

None of this needs extra access. The screenshot is the same whatever the scale. What changes is how the tool interprets it. Keep the same habits as always: invoke the helper only when you need it, and close anything private first.

A short test routine you can run yourself: open something with a clear target, such as a button with a distinctive label. Ask the assistant to point at it. Then change the scaling or zoom one notch, ask again, and see whether it still lands on the button. Repeat in the other theme. If it misses at one setting and not another, you have found exactly where the conversion breaks, and you can report that precisely.

Accessibility is the deeper reason. People who need larger text or higher contrast are the ones most likely to change these settings, so a tool that fails on them fails the people who most need help.

Try it yourself

DuckyHelper is one tool that works this way: press one key in any app (fn on a Mac, Ctrl+Shift+Space on Windows and Linux), ask out loud, and it answers out loud while you keep working. Try asking it about something on your screen while your scaling or theme is set the way you actually use it, and see whether its drawing lands on the right spot. You can try it for 10 minutes with no card.

Frequently asked questions

Does scaling change what an assistant can read?

Not what it can read, but where things are. Text is easier to read at a larger scale, but positions must be converted.

Should I turn dark mode off for help?

No. A tool should work in the theme you prefer, and if it does not, that is a fault in the tool.