TL;DR:
Computer use is an assistant working through a screen like a person: look, decide, click or type, look again.
It is harder than chat because every step changes the screen, and one wrong click can have a real effect.
The safe version asks before it sends or pays, and keeps going only while it can see that it is on track.
A chat assistant can tell you how to rename a file. A computer-use assistant can open the folder and rename it. The second sounds like a small step, but it changes what can go wrong, which is why it deserves its own name.
The short answer
Computer use means an assistant is given a view of a screen and the ability to act on it: move a cursor, click, type, scroll. Instead of describing a task, it does the task in the same apps you use, one action at a time.
What happens step by step
Every computer-use system runs some version of the same loop:
Look. Capture the screen, and sometimes the structure of the page.
Decide. Choose the next single action that moves toward your goal.
Act. Click, type or scroll.
Look again. Did the screen change the way it expected? If not, adjust.
It repeats until the task is done or it gets stuck and asks you.
Walk-through: an assistant opening a page and filling a field
You say: "Open the library site and put my name in the search box."
It looks at the screen and sees a browser with a blank tab.
It decides the next action is to focus the address bar, and types the site's address.
It looks again. The library page has loaded, with a search box near the top.
It clicks the box, then looks again to confirm a cursor is blinking in it.
It types your name and looks to confirm the text appears.
Five short steps, and between each one a look. That look is the important part. Without it, a mis-click on step two would carry on silently through steps three, four and five.
Why it is harder than chat
The screen changes under it. A pop-up, a slow page or a moved button can break the plan.
Actions have effects. A reply is just text. A click can delete, send or buy.
Screens are made for people. Buttons and icons are not labeled for a program, so it has to interpret them.
Mistakes compound. Each step builds on the last.
The two ways to see a screen
A computer-use assistant can read a screen in two ways, and many use both. One is the picture: a screenshot, interpreted by looking for text, buttons and icons. The other is structure: many apps and web pages expose a description of their own controls, with names like "Search" and roles like "button". Structure is more exact when it exists, and the picture works when it does not.
Why one wrong click matters
Think about the difference between reading and acting. If a chat assistant gets a fact wrong, you can notice and ignore it. If a computer-use assistant clicks Delete instead of Archive, the effect is already real. That is why careful systems limit what they do without asking, check the screen after each action, and stop when the screen does not look as expected.
What makes a good task for it
Steps you could describe in a few sentences.
An end state you can check by looking.
Actions that can be undone, or that ask first.
Repeated work, where a few minutes of setup saves time later.
Tasks that are vague ("sort out my inbox") or high stakes with no way to review are poor first choices. Start with something small and watch it work.
How to judge a computer-use assistant
Watch what it does when something unexpected happens. Open a pop-up in the middle of a task and see whether it notices, closes it, and carries on, or whether it keeps clicking where the button used to be. A careful assistant looks after each action. A careless one assumes.
Does it tell you what it is about to do?
Does it stop at actions that cannot be undone?
Does it say plainly when it is stuck, instead of trying random clicks?
Can you stop it at any moment?
These questions work for any product, whatever its name. The answers matter more than a long list of things it claims it can do.
What it can't do
It does not understand your intent beyond what you said. It cannot be trusted to guess which of two similar buttons you meant, and a well-built one will stop and ask. It can be slow, because it looks between steps. And it should not be handed passwords or card numbers.
Privacy and control
DuckyHelper can work in your apps in this way as an early access feature. It's built to ask first before it sends a message or pays for anything, and not to type your passwords or card numbers. Those are part of how it is built, not settings you have to remember.
Early access: Working in your apps is early access on Mac. Asking out loud, seeing your screen and drawing on it are available now.
Try it yourself
Watch for the loop in any assistant that acts on a screen: does it check the result after each action, or only at the end? Checking after each action is the sign of a careful one.
Frequently asked questions
Is computer use the same as a macro?
A macro replays fixed steps. Computer use looks at the screen and decides each step, so it can cope when the screen is different from last time.
Does it need a special app?
It works through the screen, so in principle it can use any app a person can.
Get started
