30% off your first 3 months

Get the deal

What an assistant actually sees: a screenshot, not your files

by DuckyHelperNov 12, 20264 min read
AI on your computer

TL;DR:

  • Screen-based help sees what is on your display at the moment you ask: a picture, not the files behind it.

  • It can read names and numbers that are visible. It cannot see a file's contents unless that file is open on screen.

  • File access is a separate permission. Know which one an assistant is using.

When you say an assistant "can see your computer", what does that mean? Can it read every document on your drive, or only the window in front of you? The two are very different, and the difference decides what you can safely show it.

The short answer

Most screen-based assistants work from a screen capture: a picture of what your display shows at one moment. That picture contains pixels. It has no idea about the files behind the windows, the tabs you did not open, or anything hidden or scrolled out of view.

File access is a different thing. It means a program can open and read files on your disk directly, including ones you are not looking at. It needs its own permission, and a product that offers it should say so clearly.

What happens step by step

  1. You ask a question while something relevant is on screen.

  2. The assistant takes a capture of the screen, or a region of it, at that moment.

  3. It reads the picture: text, numbers, buttons, shapes, charts.

  4. It answers from what is visible, and from general knowledge.

  5. The picture is not a window onto your disk. Anything not shown stays unseen.

Walk-through: the difference between screen and disk

Say a folder window is open and lists twelve files, with columns for name, date modified and size. You ask, "Which of these is the biggest?"

With a screen capture, the assistant reads the rows. If the size column is visible, it compares the sizes you can see and answers, for example, that a video file is the largest at 480 MB while the next is 36 MB. That comparison is real, since 480 is greater than 36 (480 ÷ 36 ≈ 13.3).

Now suppose the size column is hidden. The picture only shows names and dates. The assistant cannot tell which file is biggest, and an honest one will say so and suggest turning the column on. It will not look up sizes from the disk, because it never touched the disk.

Next you ask, "What is in the report file?" The capture shows only a file name, report_final_v3.docx. The assistant can tell you what the name suggests, but not what the document says. If you open the report and it fills the screen, it can read what is visible, and it can read more as you scroll, because each scroll changes what is on screen.

That is the whole difference in one folder: screen capture sees what is displayed, file access would see what is stored.

A second example: a spreadsheet with 500 rows, where only the first 30 fit on screen. A capture holds those 30 rows. Asking for the total of the whole column therefore gets a total of what is visible unless you scroll, select the column, or the product reads the file itself. Always check which rows an answer was based on.

Screen capture vs file access

Question

Screen capture

File access

What it sees

Pixels showing right now

Files and folders on the disk

Hidden or off-screen content

Not seen

Can be read

Typical permission

Permission to record or capture the screen

Permission to read files or folders

Privacy question

What is visible when I ask?

What is stored anywhere on my disk?

Good for

Explaining what you are looking at

Searching or processing many files

What it can't do

  • Read a file it was not shown. A name on a list is only a name.

  • See what is behind a window. Covered windows and other desktops are not in the picture.

  • Be exact on tiny text. Small or blurry text may be misread.

  • Remember what it saw, unless the product keeps a record on purpose.

Privacy and control

The practical privacy rule for screen capture is simple: what is visible is what is shared. Before you ask, close the email, the banking page or the message thread that has nothing to do with the question. A notification that pops up can end up in the picture too.

Operating systems usually ask your permission before an app can capture the screen, and let you turn that off. Check which apps hold it in your system settings. DuckyHelper looks at your screen only while the pill is open and for that answer, and it does not look when idle.

Features that act in your apps are early access. Acting is not the same as reading files, and any product should tell you which one it does.

Try it yourself

Open a folder with some files and hide one column. Ask an assistant a question that depends on that column and see what it says. Then show the column and ask again. You are watching the line between screen and disk, and you will remember it the next time you wonder what an assistant really knows.

Frequently asked questions

Can a screen-based assistant read my other windows?

Only if they are visible in the capture. Windows covered by others or on another desktop are not.

How do I know which permissions an app has?

Your operating system's privacy settings list which apps may record the screen or read folders. Check them occasionally.

Get started

DuckyHelper can
do more

It sees your screen and works in any app you use. Try it for 10 minutes, no card.

Get it for Mac, Windows or Linux

Share

Questions and jobs

Do anything

Ask it a question or hand it a job, in any app.

Doing jobs for you is in early access on Mac. Asking works on Mac, Windows and Linux.