About the role
Hourly remote project with the Expert talent network: review AI-drafted shopping and delivery tasks, their grading rubrics and agent attempts, fix unfair criteria and flag tasks that cannot be saved.
What you will do
For each task, check whether the needed information was available, whether the rubric grades what the prompt actually asked, whether a valid answer would have been marked wrong and whether the rubric follows the guidelines. Then correct the criteria, flag broken tasks and add a short note on each decision. The listing mentions about 15 hours per week.
Experience
Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects in your application), the ability to follow a long agent transcript and spot where it failed, and precise writing that checks the data rather than trusting the reference answer.
Selection process
Apply on the official Expert talent network page. The listing describes the selection steps; no application was submitted by OpenJob.
Expert talent network (Mercor): pay, legitimacy and all open roles →
Similar open jobs
- AI Conversation Evaluator · micro1 · $40-$70/hour · Remote
- QC Packaging Engineer · Turing · Not stated on the listing · 8 countries
- AI Assistant Rater (US based) · Turing · Not stated on the listing · United States
- Tooling Engineer · Turing · Not stated on the listing · Remote
- Quality Analyst-Multimodality · Turing · Not stated on the listing · Remote
- AI Trainer, Data Annotation Specialist (Safety) · Turing · Not stated on the listing · Remote
Eligibility
Remote (as stated on the listing); no country requirement listed
Flexible remote tasks
Fluent in English? Outlier's Project Aether pays for evaluating AI images, video and chat replies. No degree or technical background needed, and applying is free.
How Aether works and how to pass the test →