TL;DR:
Site actions only work where there is a web page. Desktop apps need another approach.
The fallback is to look at the screen and use a cursor like a person would.
It is slower than a saved action but works in apps that have no web version.
Plenty of the software you use has no web page at all: a desktop video editor, an old accounting program, a design tool. Saved site actions have nothing to hook onto there. The assistant needs a way that works anywhere, and the only thing every app has is a screen.
The short answer
When there is no web surface to learn, the assistant falls back to what a person does: look at the screen, find the control, move a cursor, click or type, and look again. It is slower than a saved action but universal.
What happens step by step
Capture the app window.
Find the control that fits the next step by its label, icon or position.
Move its own cursor there and act.
Look again to confirm the app changed as expected.
Repeat until done, or stop and ask.
Walk-through: a desktop app with no web version
You have a desktop invoicing program. You say: "Export this month's report to a PDF."
It sees the main window with a menu bar: File, Edit, View, Reports.
It opens Reports and looks at the list. The one named Monthly is the match.
It opens it, sees a toolbar, and finds the button whose icon and tooltip say Export.
It picks PDF in the dialog and looks again to confirm the save dialog is open.
It stops before replacing an existing file and asks you.
Count the looks: five steps, five checks. A site action would have been one call. That is the cost of the fallback.
Why keep both
Site actions are quick where they apply. The screen fallback is slower but covers everything else. Together, an assistant can say yes to far more tasks than either alone.
How it finds the right control
Without a web page to inspect, the assistant relies on what it can see: the text of a label, the icon on a button, where the control sits in the window. A button named Export is easy. An icon with no text is harder, and a good assistant uses the tooltip or the surrounding labels to confirm what it is before pressing it.
Making the fallback go well
Tell it the app and the window, so it does not start in the wrong place.
Keep the window large and unobstructed so it can see the controls.
Choose a task you can check by looking, like a file appearing.
For anything that overwrites or deletes, expect it to ask first.
Speed, honestly
The fallback takes more steps and so more time. That is a fair price for being able to work in an app at all. If a task is something you do often and the app also has a web version, a saved site action may be the better choice.
Choosing which approach to use
Think of it as a short decision. If the task is on a web page you use often, a saved site action is the first choice. If the page is unfamiliar or has changed, the assistant looks at the page again. If there is no page at all, it uses the screen with a cursor. Each step down is slower and more general than the one before.
Web page, used before: saved action, quickest.
Web page, new or changed: learn it again, then act.
No web page: use the screen and a cursor.
Good products make this choice for you, and tell you which path they took when it matters.
What it can't do
Work in what it cannot see. A window hidden behind others, or a locked screen, is out of reach.
Be fast. It waits for the app between each step.
Cope with surprises as well as you. An unexpected dialog may stop it.
Privacy and control
Working in desktop apps this way is part of DuckyHelper's early access agent features, with its own cursor while you keep using yours. It's built to ask first before it sends a message or pays for anything, and not to type your passwords or card numbers.
Try it yourself
List the apps you use that have no website. Those are the ones where the screen fallback matters. Start with a task that is easy to undo.
A last practical note: keep your hands off the app while the assistant is working in it, unless you are using an arrangement where it has its own window and cursor. Two people clicking in one window is how mistakes happen, and that is as true of a person and an assistant as of two people.
If you do have a choice, a good first test is a task that only reads or exports. Look at the result, and if it is right, try something that changes more. This ramps up trust gradually and keeps any early mistake small.
Finally, write down what a finished task looks like before you start, for example "a PDF named report in my Documents folder". With a clear end state, you can check the result in seconds, and the assistant knows when to stop instead of clicking around looking for more to do.
Frequently asked questions
Does it work in every app?
In principle, any app that shows on a screen. In practice, results vary with the app.
Why not always use the fallback?
Because it is slower and has more steps to go wrong than a saved action.
Get started
