Abstract. Human work is not a video classification problem. A useful robot skill must connect what the system perceives with what it does, how contact evolves and how the task changes when the world does not match the demonstration. This note outlines a practical pipeline for learning from human expertise while preserving evaluation, safety and data governance as first-class engineering requirements.
Physical skill is closed-loop.
A skilled worker does not execute a fixed sequence blindly. They observe geometry, material state, tool response and task progress; act; then update the next action from the result. The learning problem is therefore sequential and partially observable.
At time t, a useful observation may combine oₜ = {RGB, depth, pose, joint state, force/torque, touch, task context}. The policy produces an action or short action horizon aₜ:ₜ₊ₕ. The engineering challenge is not only fitting the mapping π(a|o); it is preserving timing, contact, uncertainty and recovery information that make the mapping valid outside a single recording.
Demonstrations need synchronized context.
Human demonstration can be recorded through task-facing cameras, depth sensing, wearable motion capture, instrumented tools, gloves or teleoperation. No sensor is universally sufficient. The right configuration depends on the task's information bottleneck.
- time-synchronised observations and actions;
- camera calibration and coordinate frames;
- task intent, object and environment metadata;
- success, intervention and failure annotations;
- consent, lawful basis, access control and retention policy.
High-quality negative and recovery data are especially important. A dataset containing only successful expert trajectories can teach nominal behaviour while leaving the system unprepared for the states created by its own small errors.
From imitation to action generation.
Behaviour cloning provides a direct baseline: predict expert actions from observations. For dexterous and multimodal tasks, the action distribution may contain several valid solutions. Diffusion Policy showed how conditional denoising can represent multimodal robot action sequences and use receding-horizon control for visuomotor tasks.
Vision-language-action models add semantic priors and language conditioning. The π₀ work, for example, builds a flow-matching action model on a pretrained vision-language backbone. These architectures are promising, but they do not remove the need for embodiment-specific calibration, safety constraints, latency control and task-level validation.
Diversity is part of the model.
Generalization depends on variation in scenes, objects, operators, viewpoints, robots and task language. Open X-Embodiment combined more than one million real-robot trajectories across 22 robot embodiments in a common format. DROID collected 76,000 trajectories across hundreds of real scenes. Both efforts support the same engineering lesson: broad, structured experience can improve transfer, while narrow data encourages brittle shortcuts.
Quantity alone is insufficient. Dataset composition should be designed around the deployment distribution and its edges: lighting, clutter, object variants, wear, occlusion, human interruption and changes in initial state. Evaluation splits must prevent near-duplicate scenes from disguising memorization as generalization.
Autonomy should expand through gates.
A learned policy should move from replay and offline metrics to simulation, instrumented lab trials and finally a bounded operational pilot. Each stage needs explicit entry and exit criteria.
Completion rate, quality and cycle time.
Performance across controlled variation.
Detection, retry and safe escalation.
Constraint violations and intervention margin.
Human oversight is not a weakness in this pathway. It is a mechanism for collecting informative edge cases while keeping the operating envelope controlled. The autonomy boundary can expand only when evidence supports it.
A reusable skill layer for real work.
GappAI's research direction is to connect expert-data capture, multimodal policy learning, embodiment adaptation and deployment assurance in one engineering loop. The long-term objective is not a separate script for every task, but transferable physical skills that can move across compatible robots and improve through governed real-world experience.
This is a research direction, not a report of completed product performance. Future GappAI publications will distinguish hypotheses, experimental results and deployed evidence explicitly.
- Open X-Embodiment Collaboration. “Open X-Embodiment: Robotic Learning Datasets and RT-X Models.” ICRA 2024.
- Khazatsky et al. “DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset.” RSS 2024.
- Chi et al. “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.” RSS 2023.
- Black et al. “π₀: A Vision-Language-Action Flow Model for General Robot Control.” 2024.