Evaluate and compare the quality of responses from multiple AI chatbots across real-world small business use cases. Main tasks listed: Create realistic business-related prompts based on defined user goals; Interact with multiple AI chatbots (max. 5 turns per conversation); Assess response quality across clarity, usefulness, and accuracy; Provide structured feedback and comparative evaluations.
As an AI Quality Analyst, you will evaluate a new personalization feature for Gemini. You will assess how well the model uses information from your past Gemini conversations, Gmail, Google Search, and YouTube activity to make responses more relevant and helpful. This role requires a unique blend of creativity and analytical rigor. You will actively design prompts from the perspective of your own personal experiences. This work may involve recording or sharing personal data (for example voice, video or account activity): read the consent and privacy terms on the official page before applying.
We are looking for experienced Software Engineers to help evaluate, benchmark, and improve the coding capabilities of frontier AI models. In this role, you will assess AI-generated code, validate solutions against real-world software engineering tasks, identify correctness and quality issues, and contribute to the development of high-quality evaluation datasets and benchmarks.
We are seeking Board Game Reasoning Experts to help train and evaluate next-generation AI models. In this role, you will leverage your expertise in board games, game mechanics, strategic reasoning, probability, and complex rule systems to create, review, and evaluate high-quality datasets that improve AI performance.
We are seeking Senior Accounting & Finance Professionals with hands-on experience in QuickBooks Online (QBO) and Omni Chat support. This role involves handling both general help inquiries and complex, data-heavy accounting and finance questions that require strong analytical and advisory capabilities. Main tasks listed: Analyze financial data and provide accounting guidance; Handle complex accounting, bookkeeping, and finance-related questions; Provide advisory-based recommendations using financial data.
We are seeking a highly qualified Music Domain Expert to evaluate and improve Large Language Models (LLMs). You will design challenging prompts, assess factual accuracy and reasoning, and identify knowledge gaps within music theory, genres, composers, artists, production, instruments. Main tasks listed: Create advanced prompts covering music theory, genres, composers, artists, production instruments; Evaluate AI-generated responses for factual accuracy, reasoning quality, completeness, and nuance; Identify hallucinations, logical inconsistencies, outdated information, and edge cases; Develop benchmark datasets and adversarial test cases.
We are seeking a highly qualified Art Domain Expert to evaluate and improve Large Language Models (LLMs). You will design challenging prompts, assess factual accuracy and reasoning, and identify knowledge gaps within fine arts, visual arts, architecture, design, art history, museums, and artists. Main tasks listed: Create advanced prompts covering fine arts, visual arts, architecture, design, art history, museums, and artists; Evaluate AI-generated responses for factual accuracy, reasoning quality, completeness, and nuance; Identify hallucinations, logical inconsistencies, outdated information, and edge cases; Develop benchmark datasets and adversarial test cases.
Job Title: Mid-Senior Azure DevOps Engineer (LangGraph Experience) We are seeking a Mid-Senior Azure DevOps Engineer with strong experience in Azure DevOps and hands-on expertise in LangGraph to support the deployment, automation, and management of AI-powered applications. The ideal candidate will have experience building scalable CI/CD pipelines, managing Azure cloud infrastructure, and deploying LLM-based agentic applications. Main tasks listed: Design, implement, and optimize CI/CD pipelines using Azure DevOps; Deploy, configure, and maintain applications and AI workflows on Microsoft Azure; Build, deploy, and support LangGraph-based agentic AI applications; Automate infrastructure provisioning using Terraform, Bicep, or ARM templates.
We are seeking a highly qualified Sports Domain Expert to evaluate and improve Large Language Models (LLMs). You will design challenging prompts, assess factual accuracy and reasoning, and identify knowledge gaps within global sports, leagues, athletes, tournaments, rules, statistics, analytics. Main tasks listed: Create advanced prompts covering global sports, leagues, athletes, tournaments, rules, statistics, analytics; Evaluate AI-generated responses for factual accuracy, reasoning quality, completeness, and nuance; Identify hallucinations, logical inconsistencies, outdated information, and edge cases; Develop benchmark datasets and adversarial test cases.
We are seeking a highly qualified Politics Domain Expert to evaluate and improve Large Language Models (LLMs). You will design challenging prompts, assess factual accuracy and reasoning, and identify knowledge gaps within political science, governance, elections, public policy, international relations. Main tasks listed: Create advanced prompts covering political science, governance, elections, public policy and international relations; Evaluate AI-generated responses for factual accuracy, reasoning quality, completeness, and nuance; Identify hallucinations, logical inconsistencies, outdated information, and edge cases; Develop benchmark datasets and adversarial test cases.