When there is no website: working with its own cursor

TL;DR:

• Where a site or app has no web side to learn from, DuckyHelper falls back to seeing and using the screen.

• It works with its own cursor while you keep using yours.

• It is slower than calling a site's actions but works in more places. This is early access.

In how DuckyHelper learns a website once and reuses it we described the fast route: turning a site's buttons and forms into actions it can call. That route needs a website. This post is about everything else, and it is part of our early access work.

Not everything is a website

Much of what people do on a computer happens outside a browser. A spreadsheet open in a desktop program, a design tool, a settings window, an older piece of software your workplace relies on. These have no pages to learn from and no site actions to call. If an assistant could only work through websites, it would be useless for a large part of the day.

We did not want that gap. DuckyHelper is supposed to be an assistant for the computer, not only for the web, so it needs a way to work where there is nothing to call.

Seeing and using the screen

The fallback is the same thing a person does. DuckyHelper looks at the screen, works out what is on it, and uses the app: it moves a pointer, clicks, scrolls and types in ordinary fields. If it can see it and use it, it can work in it.

This is the more general method. It does not depend on how an app is built, which is why it can handle software nobody planned for. It is also the reason we keep it as a fallback and not a first choice for websites, where learned actions are faster and break less.

Its own cursor, your own cursor

When DuckyHelper uses an app this way, it has a cursor of its own. You can see it moving in the window it is working in, marked with a small DuckyHelper tag. Your cursor is still yours. You can carry on with something else while it works, because it is not borrowing your mouse.

That separation matters. If an assistant took over your pointer, you would have to sit still while it worked. With its own cursor in its own window, the two of you can work at the same time. We talk about how that works in working beside you.

Slower but universal

The trade-off is plain. Looking at the screen after each step takes more time than calling an action, and it depends more on the screen being as expected. A button that moves or a dialog that pops up can throw it. In exchange, it works almost anywhere.

We think that is the right way round. Fast where a fast route exists, and general where it does not. You get the benefit of the fast route without losing the general one.

The rules are the same

Using the screen does not loosen any rule. DuckyHelper is built to ask first before it sends a message or pays for anything, and not to type your passwords or card numbers, in any app. If an app needs you to sign in, you do that part yourself and it carries on.

An example

Say you keep a household budget in a spreadsheet program on your computer, and you ask DuckyHelper to add up each category and write the totals in column D. There is no website here, so no actions to call. It looks at the sheet, finds the columns, moves its own cursor to the first empty total cell and fills it in, then moves to the next. (This is early access.)

From your side it looks like a careful person at a second keyboard. You can watch if you like, or carry on in another window. When it finishes, the ring on the edge of your screen turns green.

What helps it do well

Because it works from what it sees, it does best when the screen is clear: labelled columns, readable text, a window that is not covered by other windows. If it cannot tell what a control does, the safe course is to stop and say so, and that is what we want it to do.

It also means you can help. If a job goes astray, a short instruction such as "the totals go in the grey column" can clear up the misunderstanding, since you and it are looking at the same screen.

What this means for you

You do not have to know which route DuckyHelper will take. Ask for the job, and it picks the learned actions where it has them and the screen where it does not. If you want to see how a job went, look at the plain steps it shows.