About the role
Hourly remote project with the Expert talent network: audit the work of reviewers on AI shopping and delivery tasks and judge whether their rubric failures are fair.
What you will do
Audit complete tasks (prompt plus grading rubric): check whether the information was available, whether the rubric grades what the prompt asked, whether a valid answer would have been marked wrong and whether the rubric follows the guidelines. Then give feedback to reviewers on prompt and rubric issues. The listing mentions about 15 hours per week.
Experience
Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects when you apply), experience giving feedback to writers or reviewers, consistent availability and precise writing that checks the data instead of trusting the reference answer.
Selection process
Apply on the official Expert talent network page. The listing describes the selection steps; no application was submitted by OpenJob.
Expert talent network (Mercor): pay, legitimacy and all open roles →
Similar open jobs
- AI Agent Tester · micro1 · $25-$45/hour · Remote
- Generalist (US Only) · micro1 · $21-$45/hour · United States
- AI Evaluation Rubric Reviewer · Expert talent network · $40-$75/hour · Remote
- AI Conversation Evaluator · micro1 · $40-$70/hour · Remote
- QC Packaging Engineer · Turing · Not stated on the listing · 8 countries
- AI Assistant Rater (US based) · Turing · Not stated on the listing · United States
Eligibility
Residence in Canada, United States, Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Guyana, Paraguay, Peru, Uruguay or United Kingdom (official listing data)
Flexible remote tasks
Fluent in English? Outlier's Project Aether pays for evaluating AI images, video and chat replies. No degree or technical background needed, and applying is free.
How Aether works and how to pass the test →