How an AI explains a diagram it can see

TL;DR:

• An assistant looking at a diagram does three jobs: find the parts, read the labels, and work out the connections between them.

• The connections (arrows, lines, position) are where the explanation lives, so a good answer follows the path, not the list.

• It can misread small print or an unusual layout, so check its answer against the page and ask it to point at what it means.

You are studying a labeled diagram of the heart for a biology quiz. There are twelve labels, six arrows and a lot of red and blue. You can read each word, but you cannot yet say how blood actually travels through the picture. That gap, between knowing the names and seeing the story, is exactly what a good explanation of a diagram has to close.

The short answer

When an assistant can see your screen, a diagram arrives as an image. It does not "know" the heart from the picture alone. It combines three things: the text it can read in the image, the shapes and arrows it can make out, and what it already knows about the subject. The explanation is the result of lining those up.

What happens step by step

1. Find the parts. It separates the drawing into regions: four chambers, some large vessels, a few valves.

2. Read the labels. Each word is matched to the part it touches or points to. A label with a thin line to a vessel names the vessel, not the chamber beside it.

3. Follow the connections. Arrows and the way parts touch tell it which part feeds which.

4. Check against what it knows. If the picture and its general knowledge disagree, a careful answer says so rather than glossing over it.

5. Explain in an order a person can follow. Usually that means one path, from start to finish, not an alphabetical tour of labels.

Walk-through: a labeled diagram of the heart

Say your diagram shows the right atrium, right ventricle, left atrium, left ventricle, the aorta, the pulmonary artery, the pulmonary veins and the vena cava. You ask: "How does blood move through this?"

A good answer follows one loop and names each part as it goes:

1. Blood returning from the body enters the right atrium through the vena cava.

2. It passes to the right ventricle, which pumps it out through the pulmonary artery toward the lungs.

3. In the lungs the blood picks up oxygen, then returns through the pulmonary veins to the left atrium.

4. It moves to the left ventricle, the strongest chamber, which pushes it out through the aorta to the body.

Check it the way you would check your own notes: the path has two loops joined at the heart (body to lungs, lungs to body), and every chamber and vessel from the diagram appears exactly once. If one label is missing from the story, either the answer skipped it or the diagram has a part the answer misread.

Ask for the path, not the list: "Name the parts" gives you a list you already have. "Trace the blood from the body and back" makes the assistant use the arrows, which is where the understanding is.

Why pointing helps

Words about a picture are slippery. "The vessel on the upper left" could be three different things. If the assistant can draw on your screen, it can circle the pulmonary artery while it says the name, so you never have to guess which part it means. DuckyHelper does this: it sees your screen, draws on it and explains out loud, or works the problem out with you on a blackboard.

Reading arrows, colors and position

Diagrams carry meaning in more than words. Color often separates two kinds of thing: in a heart diagram, red usually marks blood carrying oxygen and blue marks blood without it. Position carries meaning too, since a diagram drawn as you would see it in a mirror puts the left side of the heart on the right of the page. An assistant that ignores this will explain the right words about the wrong side.

Arrows are the strongest clue. An arrow from the right ventricle to the pulmonary artery says "this feeds that", and it also says which way. If an arrow has no label, a good explanation says what it takes the arrow to mean instead of presenting a guess as a fact.

Asking better questions about a diagram

• Ask for a path: "Trace the blood from the body to the lungs and back."

• Ask for a comparison: "What is different between the left and right ventricles in this picture?"

• Ask for a why: "Why is the wall of this chamber drawn thicker?"

• Ask for a test: "Cover the labels and quiz me on the parts."

Each of these forces the explanation to use the structure of the picture, not just its vocabulary. The last one is the most useful for studying, because recalling a label with the label hidden is what an exam asks you to do.

What it can't do

• Read tiny or blurry print. If a label is a few pixels tall, it may guess. Zoom in on the diagram first.

• Know your teacher's version. Diagrams vary. A textbook may label a part differently from your class notes.

• Tell you the diagram is wrong. It may notice, or it may not. Treat a surprising claim as a reason to check.

• Replace the exam. It can explain; you still need to redraw the diagram from memory to know you have it.

Privacy and control

A diagram on your screen is just part of your screen. DuckyHelper looks only while the pill is open (fn on a Mac, Ctrl+Shift+Space on Windows and Linux) and for that answer. When it is idle, it is not watching.

Try it yourself

Open any diagram, press the key and ask for the path from one part to another. Then close the page and try to redraw it. Wherever you stall is the part to ask about next.

Frequently asked questions

Does it need the labels to explain a diagram?

Labels help a lot, but an assistant can often explain an unlabeled diagram from its shapes and what it knows. It is more likely to be wrong, so ask it what it is assuming.

Can it explain a diagram that is not biology?

Yes. Circuits, flowcharts and force diagrams work the same way: find parts, read labels, follow the connections.