Why DuckyHelper draws on your screen instead of describing it

TL;DR:

• DuckyHelper sees your work and shows you on it, with hand-drawn circles, arrows and short notes.

• Drawings clear when you click or scroll, so they do not sit in the way of your work.

• When words are enough, it just talks.

Imagine asking a friend for help with a form and hearing: "Click the third icon from the left, in the second row, next to the one that looks like a gear." You will probably still miss it. Now imagine the friend reaches over and taps the right icon. Pointing wins, and it is the idea behind how DuckyHelper explains things on your screen.

"Click the third icon from the left"

Describing a location in words is slow and easy to get wrong. It asks you to translate a description into a spot on the screen, and the translation is where people get lost. Screens full of similar-looking buttons make it worse.

A good teacher sitting next to you does not describe positions. They point. We wanted DuckyHelper to do the equivalent: since it already sees your screen, it should be able to show you the exact thing it is talking about.

Drawing right where you are looking

When you press the key and ask a question, DuckyHelper looks at what is on your screen and answers out loud. If the answer is about a particular part of the page, it can draw on it. The drawing lands on top of your own work, right where your eyes already are, so you do not have to move your attention anywhere else to see it.

This is the difference between an assistant that talks about your screen and one that works with it. "The mistake is on this line" is much more useful with a circle around the line.

What it draws

We kept the toolbox small on purpose:

• Circles, to say "this one".

• Arrows, to connect two things or point at something small.

• Short notes, written in a hand-drawn style, for the few words that help most.

The marks look hand-drawn because they are meant to feel like a teacher's pen and not like an interface element. They are drawn as it speaks, and they stay only as long as they are useful.

Drawings clear when you click or scroll

A mark on your screen is only helpful while it matches what is under it. If you scroll, the page moves and the mark would end up pointing at the wrong thing. So drawings clear when you click or scroll. We would rather lose a mark than leave one pointing at something it no longer describes.

It also means the drawings do not block your work. You do not have to dismiss anything; you just carry on, and the ink goes away.

When words are enough

Not every answer needs a drawing. If you ask what a word means or for the reason behind a rule, a clear spoken sentence is better than a circle on a page. DuckyHelper draws when location matters and talks when it does not.

And when an idea needs to be worked out step by step instead of pointed at, it moves to the blackboard. We cover that in why we built a blackboard into DuckyHelper.

A small example

Say you are working through a settings page and ask, "where do I turn off notifications?" A description would read: "Open the menu in the top right, choose the second item, then look for the toggle under Alerts." That is three places to look and three chances to get lost.

With drawing, DuckyHelper says the same thing in a sentence and circles the menu. When you click it and the page changes, the circle clears, and it can mark the next spot if you ask again. You never have to hold a list of directions in your head.

Why hand-drawn

We chose a hand-drawn look for the marks over crisp, rule-straight shapes. A slightly loose circle reads as a person pointing, and it feels less like an alert box and more like help. It also keeps the screen yours: the marks look like notes on top of your work and not like part of it.

Because the drawing is a way of speaking and not a layer of decoration, we keep it brief. A circle, an arrow or a few words are usually plenty. If a drawing starts to need a paragraph of labels, the explanation is better told on the blackboard.

What this means for you

You can ask about the thing in front of you without describing it. Press the key, say "what is wrong here?" or "where do I change this?", and look for the circle. The page on what DuckyHelper sees and draws has more about how it works.