30% off your first 3 months

Get the deal

Assistants and multiple monitors

by DuckyHelperNov 12, 20264 min read
AI on your computer

TL;DR:

  • With two monitors, the computer places every pixel on one shared grid, and a tool must convert between that grid and each screen.

  • Displays with different scaling make the conversion trickier, so a pointer drawn in the wrong place is a real risk.

  • When it is unclear which screen you mean, a good assistant asks instead of guessing.

You have a document on your laptop and a spreadsheet on the monitor to its right. You say "what is this number?" and the assistant circles something on the wrong screen, or circles nothing at all. Two displays turn a simple idea, point at a thing on the screen, into a small geometry problem. Understanding it explains most of the odd behaviour you will see.

One big canvas, not two screens

Your operating system treats several monitors as one large virtual canvas. Each display is a rectangle placed somewhere on it. A point on the canvas has an x and a y, measured from a shared origin, usually the top-left corner of the main display. A screen placed to the left of the main one has negative x values. One placed above has negative y.

So there is a difference between global coordinates, on the whole canvas, and local coordinates, measured from the top-left corner of one display. Any tool that draws or points must convert between them, and it must be told which display a screenshot came from.

What happens step by step

  1. You ask about something visible, perhaps while the pill is open.

  2. The assistant captures the screen, or the display you are using, and notes which display it came from and its size.

  3. It works out where in that image the thing you mean appears.

  4. It converts that position into global coordinates, using the display's offset and scale.

  5. It draws or points there, on the right monitor.

Every step can go wrong in an obvious way. A screenshot of the wrong display, a missing offset, or a scale that was not applied will each put the mark in the wrong place.

Walk-through: pointing at something on the second screen

Take an example. Your main display is a laptop whose canvas is 1440 wide and 900 tall, starting at (0, 0). A second monitor, 1920 wide, sits to its right, so it starts at x = 1440. The number you want is at local position (400, 300) on the second monitor.

  1. Look up the second display's offset: x = 1440, y = 0.

  2. Add the local position to the offset: x = 1440 + 400 = 1840, y = 0 + 300 = 300.

  3. The global point is (1840, 300).

  4. Check it: x = 1840 is greater than 1440 and less than 1440 + 1920 = 3360, so it lands inside the second monitor, as expected.

If the tool forgot the offset and used (400, 300), the mark would land at the left of the laptop screen, a long way from the number. That is the classic bug.

Scaling makes it harder

Many laptop screens use scaling, which makes text bigger by treating several physical pixels as one. A display described as 1440 wide in the shared canvas may really have 2880 physical pixels across. A screenshot comes back in physical pixels, so before drawing, a tool must divide by the scale: 2880 ÷ 2 = 1440. An external monitor may have a different scale, so each display needs its own factor. Applying one factor to both is another source of misplaced marks.

What it cannot do

  • It cannot see a display that is switched off, asleep, or not captured, so it cannot point at what is on it.

  • It can mix up two similar windows on two screens, so say which one you mean.

  • It may misplace marks right after you plug in or rearrange monitors. Wait a moment, or ask again.

  • It cannot read your mind about which screen you meant. Saying "the left one" helps.

Asking which screen

The most reliable fix is a question. "Do you mean the spreadsheet on the right monitor?" costs a second and avoids a wrong answer. An assistant should also say which screen it is looking at when it answers, so you can correct it.

Privacy and control

A second monitor often holds the things you keep half open: messages, email, a calendar. Before you invoke an assistant, check what is visible on every display, because the tool may look at more than one. A helper that looks only while you ask, and says what it saw, keeps you in control.

A practical habit for two-screen setups: keep the window you are asking about on the display you are looking at, and leave the other screen for reference material. Then the assistant, like a person glancing over your shoulder, has less to be confused about. If you move a window from one screen to the other, wait a second before you ask, so the capture reflects where it now sits.

Try it yourself

DuckyHelper is one tool that works this way: press one key in any app (fn on a Mac, Ctrl+Shift+Space on Windows and Linux), ask out loud, and it answers out loud while you keep working. It sees your screen when you ask and draws on it. With two monitors, say which screen you mean, for example "the spreadsheet on my right monitor", so the answer lands where you are looking. You can try it for 10 minutes with no card.

Frequently asked questions

Why did the mark show on the wrong monitor?

Usually the tool captured one display but drew using another display's offset or scale. Ask again and name the screen.

Do I need the same resolution on both monitors?

No. Different sizes and scales are normal, but they are exactly why a tool needs to convert coordinates per display.

Get started

DuckyHelper can
do more

It sees your screen and works in any app you use. Try it for 10 minutes, no card.

Get it for Mac, Windows or Linux

Share

Questions and jobs

Do anything

Ask it a question or hand it a job, in any app.

Doing jobs for you is in early access on Mac. Asking works on Mac, Windows and Linux.