When DuckyHelper looks at your screen, and when it doesn't

TL;DR:

• DuckyHelper listens and looks only while the pill is open, and for the answer to what you asked. When it is idle, it is not listening or watching.

• What it sees is used to answer you: to explain out loud, draw on your screen, or work it out with you on a blackboard.

• When it does work in your apps (early access, on Mac), it's built to ask before it sends a message or pays for anything, and not to type your passwords or card numbers.

The first question people ask us is a fair one. If an assistant can see your screen, is it watching all the time? For DuckyHelper the answer is no, and we want to explain exactly how it works, because the way it decides when to look is a design choice we made on purpose.

Nothing happens until you press the key

DuckyHelper sits on your computer and waits. To ask it something, you press one key: fn on a Mac, or Ctrl+Shift+Space on Windows and Linux. A pill opens, and while it is open DuckyHelper listens to what you say and looks at what is on your screen. Then it answers out loud while you keep working.

That is the whole trigger. It listens and looks while the pill is open, and for the answer to that question. When it has answered and the pill is closed, it is idle, and idle means no listening and no watching.

The key is the trigger because a key is easy to understand. You always know when it starts: the moment you press the key, and never before. Windows and Linux add one more switch: the app sees nothing on your screen until you turn on screen help and allow it.

Why it needs to see your screen at all

Most of the questions people ask while they work are about something on the screen. "What does this error mean?" "Why is this step wrong?" "Where do I click to change this?" An assistant that cannot see the screen makes you copy, paste and describe everything first. Seeing the screen removes that step.

What it sees is used to answer you. DuckyHelper can explain out loud, draw right on your screen to point at the line or button it means, or open a blackboard and work a problem through with you step by step. If you are stuck on a math problem in a PDF, it can circle the step where things went wrong instead of describing it in words.

When it does work in your apps, it's built to ask first

DuckyHelper can also do work for you in your apps. This is in early access, on Mac. You give it a job, and it works in that window with its own cursor while you keep using yours. When it is done, the ring on the edge of your screen turns green.

Doing things is different from explaining things, so we drew some hard lines:

• It's built to ask first before it sends a message or pays for anything. You see what it is about to send or pay for, and it's built to wait for your yes. For a message you can send it, edit it or deny it.

• It's built not to type your passwords or card numbers. When a site needs you to sign in, you sign in yourself.

• It works in the window you gave it. Your own cursor stays yours, so you can keep working on something else.

We think these rules are what make it reasonable to let software act for you. Most of what an assistant does is low risk: reading, sorting, filling in a sheet. The few things that are hard to undo, like a sent email or a payment, should always wait for a person.

Questions we get asked

Does it look at my screen when I am not asking anything? No. It looks while the pill is open and for that answer, and not otherwise. The privacy policy puts it this way: it sees your screen only in the pictures taken for your question, and it does not watch your screen in between.

Can I see when it is listening? Yes. It is listening while the pill is open. The small pill on your screen shows a soft glow while it listens and shows its reply as it talks.

Does it need my passwords to work in a site? No. It's built not to type your passwords or card numbers. If a page asks you to sign in, you do that part yourself, and it carries on from there.

Where to read the details

This post covers how DuckyHelper decides when to look. Our privacy policy covers what happens to data in full, and the help center explains each setting. If something there is unclear, or you think we got a rule wrong, write to privacy@duckyhelper.com.