Senior Data Annotation Jobs in Chicago
About The Role Senior data annotation jobs in Chicago focus on building and evaluating training datasets for reputed company systems, including LLM training pipelines, RLHF, and multimodal annotation. This is a remote, full-time role aligned to Chicago/US hiring requirements. What You Will Do • Plan and lead data labeling projects across NLP, computer vision, and content safety labeling • Execute RLHF and LLM evaluation tasks (pairwise ranking, rubric-based grading, prompt evaluation) • Create and iterate annotation guidelines with clear edge cases and decision rules • Run QA evaluation programs (gold sets, audits, inter-annotator agreement) to ensure guidelines compliance • Perform error analysis to reduce label noise and improve model performance • Collaborate with engineering on tooling, schema design, and dataset versioning for LLM training pipelines • Mentor annotators, run calibration sessions, and drive consistent labeling outcomes across distributed teams • Document dataset limitations, bias risks, and safety considerations to support responsible AI development Required Qualifications • 3+ years in data annotation, data labeling, or AI evaluation with measurable QA ownership • Experience with RLHF, prompt evaluation, or LLM response rating • Strong knowledge of annotation guideline design and compliance practices • Familiarity with NLP tasks (e.g., named entity recognition, classification) and/or CV annotation (boxes, polygons, segmentation) • Comfort working in remote workflows with ticketing, documentation, and dataset version control practices Compensation Competitive hourly base pay: $30–$50 per hour, depending on scope, domain complexity, and demonstrated leadership in training data quality and QA evaluation. How To Apply Apply through reputed company with a resume highlighting senior data annotation leadership, RLHF or prompt evaluation experience, and examples of training data quality improvements. Apply tot his job Apply To this Job
