TL;DR:
A multimodal assistant takes in pictures and words together and links them, so a caption can explain a diagram.
It can describe, compare and answer questions about what is in the image, but it can miss fine details.
Point at the exact part you care about, and check any number or label it reads.
For a long time, software that handled pictures and software that handled words were separate. One might label a photo "dog", and another might write a paragraph, but neither could answer "what does the arrow in this diagram mean?" Today, many assistants take in an image and text together and relate them. The term for this is multimodal.
This article explains what that means without jargon, walks through one slide, and covers where it can go wrong.
What multimodal means in plain words
A mode is a kind of input: text, images, sound. Multimodal means the assistant can work with more than one kind at once. When you show it a slide and ask a question, it takes in the picture and your words together and builds one understanding, rather than handling them separately.
Roughly, the image is turned into numbers that capture what is in it, the text is turned into numbers too, and the system learns to connect the two. That is why it can match the word "arrow" to the thing in the picture.
Walk-through: a slide with a diagram and a caption
A science slide shows a drawing of a plant with three arrows. The caption below says: "Water moves up from the roots, sunlight hits the leaves, and sugar moves down." You ask: "What do the three arrows mean?"
It sees the picture: It notes a plant with roots, a stem and leaves, and three arrows: one pointing up the stem, one pointing at the leaves from above, and one pointing down.
It reads the caption: The words list three things: water up, sunlight in, sugar down.
It links them: The upward arrow matches "water moves up", the arrow from above matches "sunlight", the downward arrow matches "sugar moves down". The caption supplies the names and the picture supplies the positions.
It answers: "The arrow going up the stem is water from the roots. The arrow from above is sunlight. The arrow going down is sugar moving from the leaves."
You verify: You check the slide: the arrows match the caption. If it had said the arrow up was sugar, you would spot the clash.
Neither the picture alone nor the caption alone gives the full answer. The picture shows direction, the caption gives the meaning, and together they make an explanation.
Where it goes wrong
Small details. Tiny text, thin lines, and crowded diagrams can be misread or skipped.
Counting and exact positions. How many items, or which one is exactly left of another, can be wrong.
Reading numbers. A digit or a unit on a label can be mistaken.
Unstated context. If the slide assumes you know something from an earlier slide, it may not know it.
A caption is not always right
Captions can be wrong, out of date, or about a different version of the picture. If the picture and the caption disagree, a careful assistant should point out the clash instead of choosing one silently. When you see a mismatch yourself, mention it. Asking "does the caption match the diagram?" is a good question in its own right, and it teaches you to look for the same problem in your own materials.
A related point: the assistant only sees what is on the screen at that moment. If an earlier slide defines a symbol, it may need you to say so.
Getting better answers
Point at the part you care about instead of asking about the whole slide.
Quote the label you mean, such as "the arrow labelled B".
Ask it to say what it sees before it interprets, so you can catch a misread.
Check any number, label or count against the image yourself.
Note: Seeing and understanding are different. An assistant might describe the picture correctly and still draw a wrong conclusion. Check the reasoning as well as the reading.
What this does not do
It does not mean the assistant has a perfect eye or a perfect memory of what it looked at. It builds an impression of the image, and that impression can be incomplete. Treat its description as a helpful first look, not a measurement.
Privacy and control
Anything visible in an image can be read. Slides may include names or private data. DuckyHelper looks at your screen only while the pill is open and for that answer, and can draw on your screen to show what part it means.
Try it yourself
Show an assistant a diagram from your notes and ask it to describe the diagram before explaining it. Compare its description with the original. Where they differ, you have found a limit, and learned where to look when it matters.
Frequently asked questions
Does the caption help the assistant read the image?
Yes. Text gives meaning to shapes, and the shapes show positions. Having both usually gives a better answer than either alone.
Why does it sometimes miss small labels?
Fine detail can be lost when an image is shrunk. Zoom in on the area you care about.
Can it read video too?
Some systems handle video as a series of images. Treat it with the same care.
Get started
