YO HR Consultancy

AI Data Quality Lead – Robotics &amp; Video Annotation | Remote
About the Role
We are looking for an experienced AI Data Quality Lead to own the quality standards for egocentric training data used to develop and evaluate physical-AI and robotics models.
You will be responsible for building and maintaining the golden sets, annotation SOPs, rubrics, calibration systems, quality frameworks, and audit processes that ensure high-quality ground-truth data. You will also work closely with engineering and product teams to develop annotation workflows for hand tracking, object identification, and action labeling.
This is a ground-floor opportunity to build a 0→1 data quality system at a fast-growing, well-funded startup. You will not simply inherit an existing process—you will help establish the quality standards, workflows, and systems from the ground up.
Key Responsibilities
<ul>
<li>
Own the end-to-end quality framework for egocentric training data, including:
<ul>
<li>
Golden and benchmark datasets
</li>
<li>
Acceptance criteria
</li>
<li>
Frame-level tolerance thresholds
</li>
<li>
Sampling strategies
</li>
<li>
Audit and quality-control loops
</li>
</ul>
</li>
<li>
Author, maintain, and version annotation SOPs, rubrics, and edge-case guidelines as project specifications evolve.
</li>
<li>
Translate ambiguous requirements into precise, actionable instructions for annotators and reviewers.
</li>
<li>
Establish and manage annotator training, certification, and calibration programs.
</li>
<li>
Run calibration sessions and adjudicate disagreements between annotators.
</li>
<li>
Measure and report quality using metrics such as:
<ul>
<li>
Inter-Annotator Agreement (IAA)
</li>
<li>
Kappa/alpha
</li>
<li>
Tolerance-based agreement for temporal boundaries
</li>
<li>
Acceptance/rejection rates
</li>
<li>
Rework rates
</li>
<li>
Cost per accepted hour
</li>
</ul>
</li>
<li>
Identify root causes of quality issues and determine whether they result from guideline ambiguity, annotation errors, or workflow issues.
</li>
<li>
Design human-in-the-loop workflows where AI/model-generated labels are reviewed, corrected, and escalated by human annotators.
</li>
<li>
Audit auto-generated labels and improve annotation throughput without compromising quality.
</li>
<li>
Partner with engineering and product teams to develop new annotation workflows for:
<ul>
<li>
Hand tracking
</li>
<li>
Object identification
</li>
<li>
Action labeling
</li>
</ul>
</li>
<li>
Define label schemas, annotation-tool requirements, QC dashboards, and pilot-validation processes.
</li>
<li>
Validate annotation workflows through pilot programs before scaling them into production.
</li>
<li>
Work with ML teams to understand how annotation quality impacts model performance.
</li>
<li>
Contribute to tooling and workflow improvements where required.
</li>
</ul>
Required Skills &amp; Qualifications
<ul>
<li>
5+ years of hands-on video annotation experience, including at least 1+ year working with egocentric/first-person video for robotics or embodied AI.
</li>
<li>
3+ years of experience in a lead or QA capacity, including annotator calibration, disagreement adjudication, and ownership of annotation guidelines.
</li>
<li>
Demonstrated experience building 0→1 annotation programs and taking them from pilot through production.
</li>
<li>
Strong command of annotation-quality methodologies, including:
<ul>
<li>
IAA frameworks
</li>
<li>
Kappa/alpha
</li>
<li>
Gold-set creation
</li>
<li>
Sampling strategies
</li>
<li>
Tolerance thresholds
</li>
<li>
Acceptance criteria
</li>
</ul>
</li>
<li>
Ability to design quality systems and explain the practical difference between quality targets such as 95% and 99% accuracy.
</li>
<li>
Proven experience authoring, rather than simply following, annotation SOPs and rubrics.
</li>
<li>
Ability to translate ambiguous specifications into clear and actionable annotation guidelines.
</li>
<li>
Hands-on experience with at least 2 professional annotation platforms, such as:
<ul>
<li>
CVAT
</li>
<li>
Labelbox
</li>
<li>
Encord
</li>
<li>
V7
</li>
<li>
Label Studio
</li>
<li>
Equivalent annotation tools
</li>
</ul>
</li>
<li>
Experience configuring label schemas and QA workflows within annotation platforms.
</li>
<li>
Experience auditing model-generated or automatically generated labels.
</li>
<li>
Experience designing human-in-the-loop annotation workflows.
</li>
<li>
Strong written and verbal English communication skills.
</li>
<li>
Strong analytical skills and proficiency with Excel and/or Google Sheets for quality reporting.
</li>
<li>
Strong understanding of ML concepts and how annotation quality affects model performance.
</li>
<li>
Data-driven and process-oriented mindset with strong ownership.
</li>
<li>
Comfortable working in ambiguous, fast-moving, startup environments.
</li>
<li>
Strong ability to work collaboratively with cross-functional and remote teams.
</li>
</ul>
Preferred / Bonus Qualifications
<ul>
<li>
Background in robotics, mechanical engineering, mechatronics, or computer vision engineering.
</li>
<li>
Experience working at a leading robotics company, AI/robotics startup, or similar project.
</li>
<li>
Experience managing distributed or global annotator workforces or external annotation vendors.
</li>
<li>
Experience with multimodal annotation, including:
<ul>
<li>
3D/depth data
</li>
<li>
Joint pose
</li>
<li>
Grasp outcome classification
</li>
</ul>
</li>
<li>
Experience training or fine-tuning autolabeling models.
</li>
<li>
Experience partnering closely with ML teams responsible for automated labeling.
</li>
<li>
Experience building annotation tooling or working closely with annotation-tooling engineering teams.
</li>
<li>
Experience with physical AI, embodied AI, computer vision, or robotics datasets.
</li>
</ul>
Why This Role?
The quality bar is the product.
The ground-truth data produced through this role will become a benchmark against which physical-AI models are measured. You will have direct ownership of the systems that determine whether that data meets the required quality standards.
This is a ground-floor opportunity at a fast-growing, well-funded startup where you will build the quality system from scratch rather than inherit an established process.
Role Highlights
<ul>
<li>
Role: AI Data Quality Lead – Robotics &amp; Video Annotation
</li>
<li>
Work Type: Remote
</li>
<li>
Engagement: [Contract / Full-Time – specify as applicable]
</li>
<li>
Focus: AI Training Data, Robotics, Embodied AI &amp; Video Annotation
</li>
<li>
Experience: 5+ years relevant experience
</li>
</ul>