Introduction
Most conversations about embodied intelligence focus on the model — the architecture, the policy network, the simulation engine. Far fewer focus on the thing that actually determines whether any of it works: the data the model learns from.
That’s the part we build.
Over the past 9 years, we’ve worked on data collection pipelines that capture how humans actually move, manipulate objects, and complete tasks — and turn that into structured datasets that robotics and embodied AI teams can train on directly. This post walks through how the pipeline works and why the details matter more than they might seem to.
Why Most Existing Datasets Fall Short
Foundation models have scaled faster than the physical-world data needed to ground them. Most publicly available motion and interaction datasets share the same three limitations: they’re short clips captured in a single fixed viewpoint, with no way to reconstruct what the person was actually trying to do; they record that a hand moved, but not why — no task structure, no object-interaction context, nothing a policy can learn intent from; and they’re captured once, in one environment, and don’t transfer well to a robot operating somewhere else.
This isn’t a data-volume problem. It’s a data-structure problem. You can have a million hours of video and still not have anything a robot can learn a skill from.
How We Capture Motion: Two Systems, Seven Scenarios
We run two motion capture systems in parallel, because they solve different problems.
Vicon optical capture is what we use when precision matters more than mobility — controlled-environment tasks where we need sub-millimeter skeletal tracking and frame-accurate trajectories. Specifically, it covers three scenarios:
- Scenario 1: Fine manipulation tasks. Assembly, pick-and-place, and tool-use motions where finger joint angles and contact timing have to be exact, with no room for error.
- Scenario 2: Reference motion for simulation validation. Digital twin and physics simulation engines need a ground-truth-level motion dataset to calibrate against.
- Scenario 3: Biomechanical capture. Sports motion analysis and rehabilitation movement analysis, both of which need frame-accurate skeletal trajectories.
Xsens inertial capture is what we use when the task needs to leave the lab — full-body motion recorded in unconstrained, real-world environments where a person is moving naturally through a space rather than performing for a camera rig. Specifically, it covers four scenarios:
- Scenario 4: Everyday activity capture in real homes or offices. People completing ordinary tasks in natural living environments, not constrained by a studio’s physical footprint.
- Scenario 5: Multi-person collaboration or interaction. Tasks like warehouse picking or retail floor assistance, where multiple actors are present in the scene at once.
- Scenario 6: Large-space or outdoor movement capture. Outdoor fieldwork or long-distance movement on a factory floor — spaces too large for an optical rig’s coverage area.
- Scenario 7: Multi-site, distributed volunteer capture. When the same capture protocol needs to be replicated across different cities and volunteer populations, Xsens’s portability and fast deployment make it the better fit for scaling.
Neither system alone gives you a usable dataset. Optical capture without real-world variability gives you clean but narrow data; inertial capture without precision gives you breadth without fidelity. Which system we use — and which of the scenarios above a project actually needs covered — depends on what the downstream training task requires, which is usually the first conversation we have with a client before any capture begins.
From Raw Motion to Something a Model Can Actually Learn From
Raw skeletal trajectories are not training data. The work that makes them usable is the structuring layer we build on top. Motion gets segmented into temporal action sequences, tagged against the objects being interacted with, and organized at the task level rather than the frame level — so a model isn’t just learning “joint angles changed,” it’s learning “this is what picking up a cup and pouring looks like, including the failure cases and recovery movements.”
This is also where most of our engineering time actually goes. Capture is the visible part of the pipeline; alignment, cleaning, and structuring is the part that determines whether the dataset is usable six months later by a team that wasn’t in the room when it was recorded.
Capturing Both Sides of an Action: Egocentric and Exocentric Views
A robot learning to imitate human behavior needs to understand two things at once: what the person intended to do, and what the scene around them looked like while they did it. No single camera angle gives you both — and this is one of the more underappreciated failure points in embodied AI datasets.
Egocentric (first-person) footage is captured from a head-mounted or chest-mounted camera that moves with the actor. It records intention and step-by-step decision-making: what the person is actually looking at right before they reach for something, how their hand approaches an object, the micro-adjustments they make mid-task when something doesn’t go as planned. This is the view that tells you why a motion happened, not just that it did. It’s also the view that’s closest to what a robot’s own onboard camera will eventually see — so for imitation learning specifically, egocentric data is often the closer match to deployment conditions.
The tradeoff is that egocentric footage is noisy by nature. The camera moves with the head, so there’s constant motion blur, framing that cuts off exactly the object you most want to see, and no fixed reference point for understanding where anything is in absolute space. On its own, it tells you what someone meant to do but not where they were standing while doing it.
Exocentric (third-person) footage is captured from fixed or rig-mounted cameras positioned around the scene. It records the spatial and environmental context that egocentric footage can’t: the absolute position of every object relative to every other object, how multiple agents in the same scene interact with and around each other, the full geometry of the workspace. If a task involves more than one person, or involves an object that moves out of the actor’s field of view, exocentric footage is the only place that information exists.
The tradeoff runs in the opposite direction: exocentric footage shows you everything that happened in the room, but nothing about what the actor was attending to or why they made the choices they made. Two people performing the same physical motion can have completely different intentions, and a third-person camera alone can’t tell you which.
Why fusion matters more than either view alone. Neither perspective is sufficient on its own, and most existing datasets pick one and accept the gap. Egocentric-only datasets produce models that can imitate hand trajectories but struggle with spatial grounding — they don’t know where they are. Exocentric-only datasets produce models with good scene understanding but weak task semantics — they can describe what’s happening but not predict what comes next or why.
Fusing the two requires solving a harder problem than just recording both: every frame from the egocentric stream has to be temporally and spatially aligned against the corresponding frames from every exocentric camera, despite the two systems having entirely different geometries, frame rates, and reference coordinates. Get the alignment even slightly wrong and you end up training on data where the “intention” signal and the “context” signal don’t actually correspond to the same instant in the action — which is worse than having no fusion at all, because the model learns a false correlation instead of no correlation.
This is the step where most of our calibration and synchronization work actually goes. Done correctly, the result is a dataset where a model can learn both halves of the same action at once — what the person meant to do, and what the world around them looked like while they did it — which is closer to how a robot will actually need to reason once it’s deployed.
What the Full Pipeline Looks Like
End to end, a project moves through: scoping the capture requirements against the client’s training objective, which determines whether we use Vicon, Xsens, or both; multi-camera egocentric/exocentric video recording, synchronized with motion capture; sensor alignment and temporal synchronization; data cleaning, normalization, and structuring into task-level sequences; and delivery in formats ready for direct ingestion into training pipelines.
Every stage is built to scale — we design capture protocols that can run across multiple sites and volunteer pools without losing consistency, which matters once a project moves past a single proof-of-concept batch. Compliance documentation is built into this process rather than added afterward: data provenance, volunteer consent records, and demographic balancing requirements are scoped alongside the capture protocol from day one, so the deliverable is ready for frameworks like the EU AI Act rather than retrofitted to meet them after collection is already done.
Where This Data Gets Used
The same pipeline supports a few distinct use cases, though the underlying data requirements differ more than people expect: in robotics, it supports imitation learning for humanoid robots and physical task execution models trained on real human demonstrations rather than synthetic motion; in embodied AI research, behavior modeling and action prediction systems need ground-truth human motion as a baseline; in simulation and XR, digital human animation and VR/AR interaction systems need motion data realistic enough to not break immersion; and in industrial applications, understanding human activity in retail and logistics environments usually means modeling a workflow rather than a single task.
Why This Is Harder Than It Looks
The honest version of this section: motion capture hardware is commercially available to almost anyone. The differentiator isn’t the equipment — Vicon and Xsens systems are the industry standard, not proprietary technology. It’s everything downstream of capture: synchronization across systems that weren’t designed to talk to each other, structuring raw trajectories into something with task-level semantic meaning, and doing it at a volume and consistency that holds up across hundreds of sessions and multiple physical sites.
That’s the part that doesn’t show up in a product spec sheet, and it’s the part that actually determines whether a dataset is usable.
What Sets Our Pipeline Apart
A lot of what we’ve described so far — Vicon, Xsens, egocentric/exocentric fusion — is technology any well-resourced team could assemble. What’s harder to assemble is the operational layer around it, and that’s where most of our actual differentiation lives.
Compliance is part of the pipeline, not a separate step. Most data collection teams hand over raw footage and leave provenance, consent, and demographic documentation as the client’s problem to sort out afterward. We scope compliance requirements at the same time we scope the capture protocol, so volunteer consent records, data provenance, and demographic balancing are built in from the first planning conversation — not reconstructed under deadline pressure once a regulator or downstream client asks for them.
Privacy and redaction are handled before delivery, not after a complaint. Any client-identifying or personally identifying information in supporting documentation gets systematically reviewed and removed as a standard part of the delivery process, rather than an ad hoc fix applied only when someone flags a problem. This matters more than it sounds like it should — a lot of datasets that are technically excellent become unusable for a client because the paperwork around them wasn’t handled carefully.
We coordinate across sites and demographics without losing consistency. Embodied AI datasets are only useful if they generalize, which means capture protocols need to hold up across different regions, device types, and volunteer populations — not just in a single controlled studio. Designing a protocol that stays consistent when it’s running in five different places with five different teams executing it is a logistics problem as much as a technical one, and it’s one we’ve built repeatable processes around rather than solving fresh on every project.
Capture and structuring sit under one roof. Because the same team handles motion capture, multi-view fusion, and the downstream data structuring, there’s no handoff gap where context about why a sequence was recorded gets lost between the people who captured it and the people who labeled it. That continuity is often what determines whether a dataset is still usable by a team that wasn’t in the room for the original shoot.
Where This Goes Next
Embodied intelligence is moving from a research niche to something robotics and simulation teams are actively building products around — which means the bottleneck is shifting from “can we build the model” to “do we have data good enough to train it on.” That’s the problem we’re building infrastructure for.
If you’re working on a robotics, simulation, or embodied AI project and want to talk through what a capture pipeline would look like for your specific use case, get in touch — happy to walk through the scoping process.