30% off your first 3 months

Get the deal

How an AI agent works in a window with its own cursor

by DuckyHelperNov 12, 20264 min read
AI on your computer

TL;DR:

  • The agent is given one window to work in and has a separate cursor, so your mouse and keyboard stay yours.

  • You can see what it is doing and take the window back at any time.

  • DuckyHelper's version, Drag the orb, is early access on Mac.

If an assistant moves your mouse, you cannot use your computer while it works. You are left watching, afraid to touch anything. A second cursor solves that: the agent has its own pointer in its own window, and you keep yours.

The short answer

Normally a computer has one cursor and one keyboard focus, and they belong to you. An agent that works in a window with its own cursor does not borrow either. It acts inside the window it was given, so you can write, browse or scroll in a different window at the same moment.

What happens step by step

  1. You choose the window the agent may work in.

  2. It looks at that window and decides an action.

  3. Its own cursor moves inside that window and clicks or types there.

  4. Your cursor and keyboard keep working in your window.

  5. It looks again and continues, or stops and asks.

Walk-through: an agent filling a sheet while you write an email

You have a spreadsheet of ten names and ten email addresses, and an email you want to write. You hand the spreadsheet window to the agent with the ask: "Fill the empty email column from the contact page I have open."

  • The agent's cursor works down the sheet, window one.

  • You type your email in window two. Nothing you type lands in the sheet, and nothing the agent types lands in your email.

  • You glance at the sheet now and then and see cells filling.

  • If you want it to stop, you stop it, and the window is yours again.

Two jobs finish in the time it takes to do one. Nothing in that depends on a clever trick: it depends on the agent staying in its own lane.

The window it was given

The boundary matters. An agent that is allowed to work in one window should not wander into others. A clear boundary also makes it easy to review: you know where to look.

Why a second cursor is not trivial

Computers were designed around one pointer and one place where typing goes. A second cursor means the agent's clicks and keystrokes must be directed to its window without moving yours or changing which window your typing goes to. That is a real engineering problem, and it is why not every assistant works this way. When it is done well, you do not notice it: your mouse does what you expect.

How you tell what it is doing

  • Its cursor is drawn so you can see where it is, separate from yours.

  • The window it works in shows the changes as they happen.

  • It reports when it is done, or when it needs a decision.

A visible cursor is not decoration. It is how you supervise without interrupting. If the cursor heads for the wrong button, you can see it, and you can stop it.

Writing a good hand-off

Say what the end state looks like, not just the starting move. "Fill the empty email column" is better than "work on the sheet", because it tells the agent when to stop. Name anything it should leave alone, such as the other columns. And say what to do if it cannot find something: stop and ask, rather than guess.

A small example of a good ask

Compare two requests. "Clean up my sheet" gives the agent no end state. "In the Contacts sheet, fill the empty Email cells in column C from the open contact page, leave columns A and B alone, and stop if a name has no match" names the window, the cells, the limits and the fallback.

The second takes fifteen more seconds to write and saves you from reading through a sheet to see what changed. Specific asks are also easier to check: you know exactly which cells to look at when it says it is finished.

  • Name the window.

  • Name what to change and what to leave.

  • Say how it should handle a missing value.

What it can't do

  • Work faster than the app allows. If a page takes three seconds to load, so does the agent.

  • Fix a bad request. If the contact page is missing an address, it should say so, not make one up.

  • Read your mind. Be specific about which column, which sheet and what counts as done.

Privacy and control

DuckyHelper's version is called Drag the orb, and it is early access on Mac. You drag the orb onto a window and say what you want done. It works in that window with its own cursor while you keep using yours, and the ring on the edge of your screen turns green when it is done. It's built to ask first before it sends a message or pays for anything, and not to type your passwords or card numbers.

Try it yourself

Even without an agent, picture the split: which job could you hand off if it could not touch your window? Write the ask as if for a new colleague: what, where, and how you will know it is done.

Frequently asked questions

Can the agent see my other windows?

A well-designed one works from the window it was given. Check each product's description of what it looks at.

What if I need the window back?

You can stop the agent and use the window yourself.

Get started

DuckyHelper can
do more

It sees your screen and works in any app you use. Try it for 10 minutes, no card.

Get it for Mac, Windows or Linux

Share

Questions and jobs

Do anything

Ask it a question or hand it a job, in any app.

Doing jobs for you is in early access on Mac. Asking works on Mac, Windows and Linux.