We are looking for experienced Software Engineers to help evaluate, benchmark, and improve the coding capabilities of frontier AI models. In this role, you will assess AI-generated code, validate solutions against real-world software engineering tasks, identify correctness and quality issues, and contribute to the development of high-quality evaluation datasets and benchmarks.
We are seeking Board Game Reasoning Experts to help train and evaluate next-generation AI models. In this role, you will leverage your expertise in board games, game mechanics, strategic reasoning, probability, and complex rule systems to create, review, and evaluate high-quality datasets that improve AI performance.
Job Title: Mid-Senior Azure DevOps Engineer (LangGraph Experience) We are seeking a Mid-Senior Azure DevOps Engineer with strong experience in Azure DevOps and hands-on expertise in LangGraph to support the deployment, automation, and management of AI-powered applications. The ideal candidate will have experience building scalable CI/CD pipelines, managing Azure cloud infrastructure, and deploying LLM-based agentic applications. Main tasks listed: Design, implement, and optimize CI/CD pipelines using Azure DevOps; Deploy, configure, and maintain applications and AI workflows on Microsoft Azure; Build, deploy, and support LangGraph-based agentic AI applications; Automate infrastructure provisioning using Terraform, Bicep, or ARM templates.
Compensation: $100-$150/hour, please provide a specific hourly rate expectation Availability: 20-40 hours/week, with at least 6 hours of Pacific Time overlap/day Main tasks listed: Design, refine, and iterate on rubrics and evaluation criteria for preference data; Review and label data with a high quality bar, catching subtle issues others might overlook; Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows; Synthesize findings from data work into clear write-ups, updates, and recommendations for the team.
We are looking for experienced Data Analysts (MLE Bench) to contribute to benchmark-driven evaluation projects focused on real-world machine learning systems. This role involves hands-on analytical work with production-like datasets, metrics, and ML outputs to help evaluate, diagnose, and improve the performance of advanced AI systems.
We are looking for experienced Software Engineers (SWE Bench – Data Engineer / Data Science) to contribute to benchmark-driven evaluation projects focused on real-world data engineering and data science workflows. This role involves hands-on work with production-like datasets, data pipelines, and data science tasks to help evaluate and improve the performance of advanced AI systems.
About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc. Main tasks listed: Analyze and triage GitHub issues across trending open-source libraries; Set up and configure code repositories, including Dockerization and environment setup; Evaluating unit test coverage and quality; Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc. Main tasks listed: Analyze and triage GitHub issues across trending open-source libraries; Set up and configure code repositories, including Dockerization and environment setup; Evaluating unit test coverage and quality; Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
A well-established company that is leveraging the advanced power of technology to help realize the science-fiction fantasy of collaborative and open-ended computer dialogues, is looking for Go develoeprs. The engineer will be working together on the definition, design, and delivery of new features with cross-functional teams. The company is developing the next generation of dialog agents, which will have a wide range of uses in areas including education, entertainment, and general question-answering. Main tasks listed: Review the code / solutions generated by an AI system, ensuring adherence to quality standards and best practices; Organize the development cycle, effectively manage project priorities, and specify goals and deadlines; Use your expertise in Go programming to help resolve difficult coding issues that come up during AI validation; Create a collaborative environment within the team that encourages innovation, communication, and continued improvement.
We’re looking for a ML Developer to drive the design, development, and delivery of advanced machine learning solutions. The ideal candidate is not just a strong individual contributor but also a technical leader capable of setting direction, mentoring team members, and ensuring that ML initiatives align with business goals. Competitive ML experience (e.g., Kaggle, benchmarks) is a strong plus. Main tasks listed: Own end-to-end DS/ML solution development — from data pipelines and model design to deployment and monitoring; Translate business objectives into robust ML architectures that accurately capture business logic and context; Collaborate cross-functionally with Product, Engineering, and Business stakeholders to define problem statements and success metrics; Evaluate and optimize models for performance, scalability, and accuracy using state-of-the-art techniques.