Looking Alive
An interactive, perceptive Leonardo da Vinci.
A character that doesn't just talk at you. It notices you, and has the restraint to know when not to.
…strong model implementations of the deep neural network and GMM-HMM, with clear optimizations.
The build, in five chapters
One character across four bodies, plus the idea that ties them together. Two chapters have work you can read today: The Mind is built and running, and The Face is underway. The last two are mapped. Open any chapter below.
The hard part of “alive” isn't the face, it's the judgment, and the restraint. Out of everyone in view, who is genuinely open to a moment, is now the right time, or is the honest answer to wait? That engine is built and runs today, in real time on a single consumer GPU.
It reads who is genuinely present
Graded, directional gaze and head orientation tell it who is oriented toward it and lingering, versus who is only passing through. Each person is tracked with a stable ID across a small group.
It engages with restraint — or waits
It commits to at most one person, and only when two independent signals are both there: a genuine relevance signal, and a social-license signal (they have come close, they have lingered). If either is missing, the honest output is to wait.
What it says is grounded in the real Leonardo
When it does engage, it speaks authored, fact-checked lines drawn from Leonardo's own notebooks and the historical record — always about the shared craft, and never a word about a person's appearance, face, or gaze.
Every decision — including every restraint — is auditable
Every choice, including every deliberate choice to hold back, is re-derivable from the log. You can reconstruct who it engaged, who it passed over, and why. “Why not her?” has an answer.
The live pipeline, annotated in real time: who it detects, who it commits to, who it deliberately passes over, and the line it opens with. Faces are de-identified and the expression read is removed — the benchmark footage was never consented for publication, so only the system's own reasoning is shown.
Restraint only means something if you can show what the greedy version would have done instead. So the system is built to be measured against an indiscriminate baseline that engages the nearest person every chance it gets, the “salesman.” The baseline and the selectivity metrics are implemented; the headline measured run awaits consented or synthetic footage.
A character staged for guests has to be trustworthy and tunable, so the system logs and visualizes every choice it makes: who it engaged, who it deliberately passed over, for how long, and exactly which signals drove each decision. Nothing is a black box.
Who & where. Are they oriented toward it? Have they come close and lingered?
Is there a genuine signal, and is engagement licensed right now? Commit to one person — or wait.
Open with a fact-checked line about the shared craft, in character. The reason it engaged stays in the log.
The perception layer stands on open, peer-reviewed methods, composed into a single real-time pipeline. Object naming, commenting on a specific held item, is the weakest channel and is currently gated off until higher-resolution detection can meet the project's faithfulness bar.
The live engine above is deliberately simple and auditable. The depth sits one layer down, in how the underlying models were built and stress-tested: two models, a hard question about whether the character's explanations are actually true, and an adversarial review whose job was to break all of it.
Trained honestly, and checked before it was trusted
A facial-expression deep neural network (EfficientNet-B0, transfer-learned on FER2013): ~0.63 accuracy and 0.59 macro-F1, near the ~65% human ceiling for this data, reported across five seeds with calibration checked (ECE ≈ 0.04). Before trusting a single number, I pixel-hashed the images and removed near-duplicates that straddled the train/test split — the quiet leak that inflates results if you don't look.
It failed — so I proved why, instead of burying it
Chaining expression into a temporal GMM-HMM to read engagement did not beat the floor. Rather than drop the result, I diagnosed it: a probe and a frame-shuffle control localized the failure to the features, not the idea. Swapping in head-pose and action-unit features cleared the floor, modestly but measurably — matching what the published literature finds.
Is the explanation actually faithful?
The centerpiece isn't accuracy, it's honesty. Every decision is auditable by construction: an independent re-implementation replays the log and reproduces each call frame-for-frame. A forward-simulatability test then asks whether the shown reason predicts the choice — a result I left honestly inconclusive on current footage, with a synthetic control proving the test itself works.
The expression model is gated behind an explicit consent flag and off by default. An adversarial ethics review caught it running on non-consenting pedestrians in the benchmark footage, so it was disabled — the honest cost was a weaker comparison, and the right trade. The deployed character decides on gaze, proximity, and dwell, never emotion recognition on people who did not consent.
This modeling work — the deep neural network and the GMM-HMM, their documentation, and their optimizations — earned an Academic Award of Excellence in a formal evaluation of the submission.
The submission stands out for its extensive documentation, including a detailed README, docstrings, and inline code comments, and for the strong model implementations of the deep neural network and GMM-HMM, with clear optimizations.
The PhD arc is built to bring four researchers' strengths together: a believable, deployable character that perceives and reasons about people, explainably, then steps off the screen into a physical, reactive robot.
Markus Gross
Interactive digital characters and the technology that makes them feel present, including projection into physical space.
Joseph Campbell
Theory of mind, anticipating human intent, and interpretable interaction, the backbone of the “explain every decision” principle.
Heni Ben Amor
Reactive control and robot learning: characters and robots that respond to people in the moment, the engine behind the robotic phase.
Stelian Coros
Physics-based, expressive character and robot motion, how a believable performance transfers to a body that obeys physics.
The single most repeatable bit of theme-park magic is a character who makes a guest feel seen. Today that depends on a gifted human performer. This builds it as a real-time, repeatable, explainable system: a character that notices the specific guest in front of it, reacts in persona, plays to a crowd, and eventually steps off the screen into the room. Da Vinci is the first host; the perception and decision engine is the product.
- [1]Wampfler, R., et al. (2025). A Platform for Interactive AI Character Experiences (Digital Einstein). SIGGRAPH Conf. Papers '25.
- [2]Campbell, J. & Ben Amor, H. (2017). Bayesian Interaction Primitives: A SLAM Approach to Human-Robot Interaction. CoRL, PMLR 78.
- [3]Campbell, J., Stepputtis, S. & Ben Amor, H. (2019). Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks. RSS. arXiv:1908.04955.
- [4]Oguntola, I., Campbell, J., Stepputtis, S. & Sycara, K. (2023). Theory of Mind as Intrinsic Motivation for Multi-Agent RL. ICML Workshop. arXiv:2307.01158.
- [5]Zhang, X.-J., et al. (2025). Model-Agnostic Policy Explanations with Large Language Models. COLM. arXiv:2504.05625.
- [6]Serifi, A., et al. (2024). Robot Motion Diffusion Model (RobotMDM): Motion Generation for Robotic Characters. SIGGRAPH Asia.
- [7]Coros, S., et al. (2013). Computational Design of Mechanical Characters. ACM TOG 32(4), SIGGRAPH.
- [8]Bates, J. (1994). The Role of Emotion in Believable Agents. Communications of the ACM 37(7).
- [9]Cheng, T., Song, L., Ge, Y., et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection. CVPR. arXiv:2401.17270.
- [10]Jocher, G., et al. (2024). Ultralytics YOLO11 (software).
- [11]Zhang, Y., Sun, P., Jiang, Y., et al. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. ECCV. arXiv:2110.06864.
- [12]Abdelrahman, A. A., et al. (2022). L2CS-Net: Fine-Grained Gaze Estimation.
- [13]Lin, T.-Y., Maire, M., Belongie, S., et al. (2014). Microsoft COCO: Common Objects in Context. ECCV.
- [14]Glas, D. F., Shiomi, M., Kanda, T., et al. (2017). Personal Greetings: Personalizing Robot Utterances Based on Novelty of Observed Behavior. Int. J. of Social Robotics.