TL;DR:
A spoken answer passes through five stages: speech to text, understanding the screen, writing the answer, text to speech and streaming.
Streaming lets the voice start before the whole answer exists, which is why replies feel quick.
Each stage can go wrong in its own way, so a wrong answer is not always a thinking error.
You ask a question out loud and a moment later a voice answers. It feels like one step. Underneath, your sound waves become text, the text is understood, an answer is written, and that answer is turned back into sound. Knowing the stages explains both why it works and why it sometimes goes wrong.
This article follows one question through all of them.
The five stages in one list
Speech to text. Your voice is turned into written words.
Understanding the screen. If you allow it, the assistant looks at what is on your display to know what you mean by this or that.
The answer. It writes a reply.
Text to speech. The reply is turned into a voice.
Streaming. The pieces are sent as soon as they are ready, instead of waiting for all of them.
What each stage does
Speech to text
Software listens to the sound and decides which words were spoken. It is good with clear speech and ordinary words and weaker with names, accents, jargon and background noise. Whatever it writes down is all the rest of the system ever sees: if it hears bat instead of that, no later stage knows.
Understanding the screen
Questions often point at something: what does this error mean, why is this number red. To answer, the assistant needs a picture of the screen or the text on it. This step should only happen when you ask, and it is worth knowing which tools do it and when.
The answer
The assistant combines your words and, if available, the screen contents, and writes a reply. This is the stage most people think of as the AI. It can still be wrong, so it needs checking like any other answer.
Text to speech
A voice is built from the written reply. Numbers, abbreviations and symbols have to be turned into words: 3/4 might be read as three quarters or as March fourth. Good systems choose well, but not always.
Streaming
If everything had to finish before the next stage began, you would wait for the whole answer to be written, then for the whole voice to be made. Streaming avoids that. The answer starts being turned into voice while later words are still being written. You hear the first sentence while the last is still on its way.
Walk-through: one spoken question traced to a spoken answer
This is an example, not a measurement of any product. You are looking at a spreadsheet with a cell showing #DIV/0!, and you press a key and say: why does this cell say div zero?
Speech to text. Your sound is turned into the words why does this cell say div zero. The system may write div zero or divide by zero. Either is fine here.
Understanding the screen. The assistant looks at the screen and finds the cell. It sees the formula, which divides one number by another, and sees the second cell is empty.
The answer. It writes: The formula divides by the value in the cell beside it, and that cell is empty. Dividing by zero is not defined, so the sheet shows an error.
Text to speech. The reply is turned into a voice. The error code is read out as it is spoken, as a few words.
Streaming. The first sentence starts playing while the second is still being written. You hear the start of the answer almost at once.
Now check the logic: the sheet really does show #DIV/0! when a formula divides by an empty or zero cell, so the answer is right. Your next step is yours: enter a value, or wrap the formula so it handles an empty cell.
Why a wrong answer is not always a wrong brain
Because there are five stages, a mistake could come from any of them.
The first stage misheard you, so it answered a different question.
The second stage looked at the wrong window.
The third wrote a mistake.
The fourth read a symbol the wrong way.
The fifth stuttered, so a word dropped out.
This is why it helps to see the words the system thought you said. If the question it answered is not your question, you can fix the cause.
What it can't do
It cannot hear over very loud noise. A headset microphone helps.
It cannot see what you do not share. If it has no view of the screen, say what you are looking at.
It cannot make an answer true by speaking it confidently.
Streaming cannot take back what it has already said. If the answer changes midway, you may hear a correction.
Privacy and control
In this chain, your voice and possibly your screen go somewhere to be processed. Find out where, for how long, and whether it is kept. A sound design listens only while the pill is open. DuckyHelper works this way: it only listens or looks while the pill is open and for that answer. When a tool does not say, assume more is kept than you expect and avoid sharing anything private.
Try it yourself
Ask your assistant a question about something on your screen.
Note which stage you think went wrong when an answer misses. Was it misheard? Was it the wrong window?
Say a number or a name and see how it is read back.
Try the same question with and without the screen shared and compare.
Frequently asked questions
Why does the voice sometimes start and then pause?
The next part of the answer may not be ready when the first part finishes. That is the cost of starting early.
Is the voice the same thing that thinks?
No. A separate stage turns the written answer into sound. That is why a voice can mispronounce something the text got right.
Get started
