Orion · Learn · Lesson 3 of 6

How does Orion see?

From camera pixels to a turn of the head: finding faces and hands, keeping track of who's who, following a face or your hand, and what Orion's camera never does.

Orion has one wide-angle camera in its head. On its own, a camera only produces a grid of coloured dots. This lesson follows those dots until they become something Orion can act on: “a person is over there, talking to me”, or “your hand just moved left”.

Finding faces, hands and things

Orion runs three Detector A model that finds things of one kind in a picture, such as faces, and marks each with a box and a score. on the camera picture: one for faces, one for palms and one for everyday objects. Each marks what it finds with a box and a Score (confidence) A detector's number from 0 to 1 for how sure it is. Orion only keeps finds above a set threshold.. Orion keeps only the finds that score above a threshold.

Picking that threshold is a trade-off: too strict and it misses people, too loose and it sees faces that aren’t there.

On Orion On the Pi alone, faces are checked 10 times a second. An optional AI accelerator An add-on chip that runs AI models much faster than a normal processor. Orion can use a Raspberry Pi AI HAT+. raises that to 15.

Under the hood

The CPU detectors are YuNet (faces), MediaPipe palm detection and YOLO11n (objects); with a Raspberry Pi AI HAT+, SCRFD, a palm model and YOLOv11n run on the Hailo chip. One worker schedules all three and always uses the latest frame, dropping older work. Thresholds and rates are in vision/orion_vision/detector_policy.py.

face 0.93face 0.58palm 0.81cup 0.66book 0.31
Faceskeep above 0.7510 a second (3 while you talk)
Palmskeep above 0.6010 a second (3 while you talk)
Objectskeep above 0.40once a minute
Running on
Each box is a guess with a score from 0 to 1. Too strict and Orion misses the person at the back; too loose and it sees faces in the wallpaper. The scene is an illustration; the thresholds and rates are Orion's.

From a pixel to a direction

A box in the picture only says “this face is at pixel 412”. That’s no use for turning toward it, because the camera moves with the head: turn the head and the same face lands on a different pixel.

So Orion records exactly where the camera was pointing when each picture was taken, and turns every find into a Bearing A direction measured from the lamp's base: how far left or right, and how far up or down.: a direction from the lamp’s base that doesn’t change when the head moves.

On Orion The camera’s Field of view How wide a camera sees, as an angle. Orion's camera takes in about 102 degrees side to side. is about 102° side to side. Pictures are analysed at 640 × 360 pixels.

Under the hood

Each frame carries its capture time (the middle of the exposure). The runtime supplies the camera’s pose at that instant, worked out from the joint angles and the robot description, so a picture taken mid-turn is still placed correctly. Detection works in pixels; everything after it works in bearings.

Orion, from above🙂
The camera's picture (640 pixels wide)
🙂
In the picture at x = 527Direction from the base 39° right
Drag the person, then turn the head. Their spot in the picture changes, but Orion combines it with where the camera was pointing at that instant, so the direction from the base (the blue line) stays put.

Keeping track of who’s who

A detector looks at each picture on its own. It doesn’t know that the face in this picture is the same person as the face in the last one. And sometimes it misses a face for a frame or two.

A tracker joins the finds into numbered Track A detector's finds over time, joined into one numbered person or hand, so Orion knows it's the same one.. It predicts where each person will be next, and a new find close to that prediction keeps the same number. A short miss doesn’t lose anyone.

On Orion A track survives one second without being seen. Finds within 20° of a prediction count as the same person.

Under the hood

vision/orion_vision/tracker.py keeps a constant-velocity Kalman filter per track, in bearings rather than pixels: each track has a direction and a speed, and every new find nudges both. Track numbers never restart, even after the camera is switched off and on.

Orion's rightOrion's left
Track numbers handed out so far 0
Black dots are detections this instant; orange rings are tracks. A dashed ring is predicted, not seen. A track survives 1 second unseen, and a detection within 20° of its prediction keeps its number. Push the missed detections up and watch new numbers appear.

Face tracking: choosing whom to look at

With several people in view, Orion has to pick one to look at, and then not flit between them. It sticks with its current target. It only switches to someone else who is looking at it and is clearly closer, and only once that’s been true for a moment.

Guess first Ben steps much closer but is looking at his phone. Does Orion switch to him?

🙂AdaOrion is looking here
🙂Ben
😶Cy
Slide a face closer (bigger). Orion only switches to someone who is looking at it, looks at least 1.5 times bigger than its current target, and stays that way for a full second. Otherwise it would flick between people.

Face tracking: turning to look

Knowing where to look, Orion turns the way people do: the head moves first, because it’s light and quick. A moment later the base starts turning too, slowly, and the head turns back toward the middle as the base takes over. The arm holds its pose throughout.

Tiny movements are ignored, so Orion doesn’t twitch every time someone shifts in their seat.

On Orion Orion plans a new look every 0.1 seconds. The base joins after 0.2 seconds, and anything within 4° counts as already looking.

Under the hood

runtime/src/expression/gaze.rs solves the head angles against the robot description so the camera points along the bearing, caps the head’s step per update, and compiles each look as a short anchor-relative motion: aim, hold for 1.1 seconds, then return to the anchor. The next update replaces it from the current commanded state, so the movement stays smooth.

Orion's rightOrion's left🙂
Now: looking
Base 1°Head swivel -0°Head nod -21°
Drag the face around (you're looking at Orion, so its left is on your right). The blue circle is where Orion is looking. Watch the numbers: the head swivel jumps first, then the base takes over the turn while the head swivel goes back toward the middle. A simplified copy of Orion's planner, using its real timings.

Face tracking: who’s talking?

When you say “Hey Orion” in a room with other people, Orion turns to you, not to whoever happens to be closest. It works out who spoke by watching mouths: the face detector also marks points on each mouth, and Orion compares how much each one moved while the wake word was said.

If it can’t tell, it doesn’t guess. It would rather not turn than turn to the wrong person.

On Orion The chosen speaker can stay Orion’s target between replies. If they leave, Orion won’t swap in a bystander.

😮AdaOrion turns to Ada
🙂Ben

Ada's mouth moved clearly more, so Orion takes Ada as the speaker and keeps following them for the conversation.

The score is how much the mouth points (from the face detector's landmarks) moved, minus that face's normal jitter. Orion needs at least 0.05, and a lead of 0.05 over the next face.

Hand tracking

In Hands mode, Orion looks for your palm instead of faces, and follows it. Move your hand left and the head follows, like a torch you can steer: handy for pointing its light where you’re working.

It uses exactly the same turning rules as faces: head first, base after, small movements ignored. When your hand leaves the picture, Orion holds still for a moment in case it comes back, then returns to its pose.

On Orion A hand can’t wake a resting lamp or switch the motors on. Only a confirmed “Hey Orion” can wake it.

Orion's rightOrion's left✋
Now: looking
Base 1°Head swivel -0°Head nod -21°
Drag the hand around (you're looking at Orion, so its left is on your right). The blue circle is where Orion is looking. Watch the numbers: the head swivel jumps first, then the base takes over the turn while the head swivel goes back toward the middle. A simplified copy of Orion's planner, using its real timings.

Finding something on request

You can also ask: “look at the mug”. Orion runs the object detector, finds the mug, and points its head at it. There are four possible answers, and Orion is honest about each: it found it; it found it but can’t turn that far; it found several and needs you to choose; or it can’t find one.

On Orion Answers to “look at …” are exactly aimed, clamped, ambiguous or not_found.

furthest Orion can turnmugbookbookplant

aimedFound one mug: Orion aims its head at it and holds there as a work anchor.

An illustration of the four results the real request can return. One camera can't measure distance, so Orion assumes objects sit on the desk to work out where to point.

Tracking modes and privacy

You decide how much Orion uses its camera. Four tracking modes go from “nothing in the background” to “always follow faces” or “follow my hand”. Separately, a privacy switch turns the camera off completely.

Pictures only ever live in the Pi’s memory. They’re not saved. One goes online when you ask Orion to look at something, and with each remark Orion makes in Demo mode.

On Orion Switching the camera off closes it and clears the images in memory, in every mode.

Tracking
  • ●Detects: Face detection for Orion's own behaviourson the Pi
  • ●Follows: The person talking to it, during a conversationvia the runtime
  • ●Viewing in Studio, with one cyan light on the ring while it's onneeds permission

The default: Orion notices, but doesn't stare.

Whatever the mode, the camera can't wake a resting lamp or switch the motors on, and images are never written to disk.