← Back to projects

AI / Robotics

Looking Alive

An interactive, perceptive Leonardo da Vinci.

A character that doesn't just talk at you. It notices you, and has the restraint to know when not to.

Active Research~26 FPSSingle GPU
Computer VisionGaze EstimationMulti-Person TrackingRestraint PolicyExplainable AIReal-Time

Every decision on the record: who it engaged, who it deliberately passed over, and which signals drove each call. Nothing is a black box, “why not her?” has an answer.

…strong model implementations of the deep neural network and GMM-HMM, with clear optimizations.

Kimberley Merritt · Academic Award of Excellence

The build, in five chapters

One character across four bodies, plus the idea that ties them together. Two chapters have work you can read today: The Mind is built and running, and The Face is underway. The last two are mapped. Open any chapter below.

Chapter 4 / 5

// chapter 03 · Sim-to-Real

The Bridge

Next build

Cross from simulation to hardware.

// simulation viewportimage coming

Expressive motion is easy to author in a simulator and hard to trust on real actuators. This phase builds the bridge: performances designed and tested in sim, then transferred to hardware that has to obey physics, joint limits, and safety, without losing the believability that makes it read as Leonardo.

// done looks like

A move authored in sim plays back on the hardware and still feels alive.

// what I'll build

Physics simulation of the head and neck, then the full body

Domain randomization to close the sim-to-real gap

Motion retargeting from authored performance to real actuator limits

A safety envelope that holds with guests in arm's reach

// what it'll show

// sim ↔ real split-screenplanned
// actuator tracking plotplanned
// safety envelope mapplanned

Placeholder chapter. Fills in with real captures as this phase is built.

// the research behind every chapter

The PhD arc is built to bring four researchers' strengths together: a believable, deployable character that perceives and reasons about people, explainably, then steps off the screen into a physical, reactive robot.

Markus Gross

ETH Zürich / Disney Research

Interactive digital characters and the technology that makes them feel present, including projection into physical space.

Joseph Campbell

Purdue, CAMP Lab

Theory of mind, anticipating human intent, and interpretable interaction, the backbone of the “explain every decision” principle.

Heni Ben Amor

Arizona State, Interactive Robotics Lab

Reactive control and robot learning: characters and robots that respond to people in the moment, the engine behind the robotic phase.

Stelian Coros

ETH Zürich, Computational Robotics Lab

Physics-based, expressive character and robot motion, how a believable performance transfers to a body that obeys physics.

// why it matters

The single most repeatable bit of theme-park magic is a character who makes a guest feel seen. Today that depends on a gifted human performer. This builds it as a real-time, repeatable, explainable system: a character that notices the specific guest in front of it, reacts in persona, plays to a crowd, and eventually steps off the screen into the room. Da Vinci is the first host; the perception and decision engine is the product.

// selected references

A curated selection from a maintained annotated bibliography of 60+ sources, the research grounding plus the third-party methods the build stands on.

Research grounding

  1. [1]Wampfler, R., et al. (2025). A Platform for Interactive AI Character Experiences (Digital Einstein). SIGGRAPH Conf. Papers '25.
  2. [2]Campbell, J. & Ben Amor, H. (2017). Bayesian Interaction Primitives: A SLAM Approach to Human-Robot Interaction. CoRL, PMLR 78.
  3. [3]Campbell, J., Stepputtis, S. & Ben Amor, H. (2019). Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks. RSS. arXiv:1908.04955.
  4. [4]Oguntola, I., Campbell, J., Stepputtis, S. & Sycara, K. (2023). Theory of Mind as Intrinsic Motivation for Multi-Agent RL. ICML Workshop. arXiv:2307.01158.
  5. [5]Zhang, X.-J., et al. (2025). Model-Agnostic Policy Explanations with Large Language Models. COLM. arXiv:2504.05625.
  6. [6]Serifi, A., et al. (2024). Robot Motion Diffusion Model (RobotMDM): Motion Generation for Robotic Characters. SIGGRAPH Asia.
  7. [7]Coros, S., et al. (2013). Computational Design of Mechanical Characters. ACM TOG 32(4), SIGGRAPH.
  8. [8]Bates, J. (1994). The Role of Emotion in Believable Agents. Communications of the ACM 37(7).

Methods & systems

  1. [9]Cheng, T., Song, L., Ge, Y., et al. (2024). YOLO-World: Real-Time Open-Vocabulary Object Detection. CVPR. arXiv:2401.17270.
  2. [10]Jocher, G., et al. (2024). Ultralytics YOLO11 (software).
  3. [11]Zhang, Y., Sun, P., Jiang, Y., et al. (2022). ByteTrack: Multi-Object Tracking by Associating Every Detection Box. ECCV. arXiv:2110.06864.
  4. [12]Abdelrahman, A. A., et al. (2022). L2CS-Net: Fine-Grained Gaze Estimation.
  5. [13]Lin, T.-Y., Maire, M., Belongie, S., et al. (2014). Microsoft COCO: Common Objects in Context. ECCV.
  6. [14]Glas, D. F., Shiomi, M., Kanda, T., et al. (2017). Personal Greetings: Personalizing Robot Utterances Based on Novelty of Observed Behavior. Int. J. of Social Robotics.
EOF

Joey Schnepel · Phoenix, AZ · 2026