← All jobs
Expert talent network

AI Evaluation Rubric Reviewer

English remote
$40-$75/hour
Verified 2026-11-16
Apply on the official platform

You will continue to the official application page.

The application link may earn OpenJob a commission. It does not affect how listings are ordered or assessed.

About the role

Hourly remote project with the Expert talent network: review AI-drafted shopping and delivery tasks, their grading rubrics and agent attempts, fix unfair criteria and flag tasks that cannot be saved.

What you will do

For each task, check whether the needed information was available, whether the rubric grades what the prompt actually asked, whether a valid answer would have been marked wrong and whether the rubric follows the guidelines. Then correct the criteria, flag broken tasks and add a short note on each decision. The listing mentions about 15 hours per week.

Experience

Substantial experience writing or reviewing rubrics for LLM evaluations (list the projects in your application), the ability to follow a long agent transcript and spot where it failed, and precise writing that checks the data rather than trusting the reference answer.

Selection process

Apply on the official Expert talent network page. The listing describes the selection steps; no application was submitted by OpenJob.

Similar open jobs

Browse all open remote AI jobs

Eligibility

Remote (as stated on the listing); no country requirement listed

Flexible remote tasks

Fluent in English? Outlier's Project Aether pays for evaluating AI images, video and chat replies. No degree or technical background needed, and applying is free.

How Aether works and how to pass the test →
📢 Get new remote AI jobs for your region every day on Telegram →

Get roles like this one on Telegram

This listing does not limit eligible countries, so pick the channel for where you live. Two messages a day per channel.

🌎 Join OpenJob Americas🌍 Join OpenJob Europe🌍 Join OpenJob Africa🌏 Join OpenJob Asia

​