Urgent hire: ASAP start, 4 week contract. Applications reviewed on a rolling basis, apply early. Main tasks listed: Use AI coding agents throughout development, QA, and validation; Participate in onboarding, calibration, and quality reviews; deliver to the quality bar on agreed milestones.
We are seeking a Dataset Enablement Manager to bridge the technical gap between scientific data pipelines and centralized commercial teams. In this role, you will package complex technical datasets into high-impact collateral, educate client-facing teams on domain-specific benchmarks, and drive the multi-client sell-through of our OTS dataset portfolio to leading global AI labs. Main tasks listed: Pipeline Tracking & Visibility: Maintain real-time oversight of active dataset pipelines across all frontier science, math, and scientific software engineering teams; Technical Collateral & Sample Packaging: Assemble production-grade sample packs, schema definitions, benchmark comparisons, and technical documentation to effectively showcase dataset quality; Commercial Enablement & Training: Lead regular deep-dives and training sessions for sales, account, and solutions teams to communicate value propositions, methodology, and target LLM use cases; Sell-Through Performance & Monetization: Track multi-client utilization metrics, identify under-monetized dataset assets, and drive performance reporting to maximize commercial sell-through.
We are looking for experienced Software Engineers to help evaluate, benchmark, and improve the coding capabilities of frontier AI models. In this role, you will assess AI-generated code, validate solutions against real-world software engineering tasks, identify correctness and quality issues, and contribute to the development of high-quality evaluation datasets and benchmarks.
We are seeking Board Game Reasoning Experts to help train and evaluate next-generation AI models. In this role, you will leverage your expertise in board games, game mechanics, strategic reasoning, probability, and complex rule systems to create, review, and evaluate high-quality datasets that improve AI performance.
Job Title: Mid-Senior Azure DevOps Engineer (LangGraph Experience) We are seeking a Mid-Senior Azure DevOps Engineer with strong experience in Azure DevOps and hands-on expertise in LangGraph to support the deployment, automation, and management of AI-powered applications. The ideal candidate will have experience building scalable CI/CD pipelines, managing Azure cloud infrastructure, and deploying LLM-based agentic applications. Main tasks listed: Design, implement, and optimize CI/CD pipelines using Azure DevOps; Deploy, configure, and maintain applications and AI workflows on Microsoft Azure; Build, deploy, and support LangGraph-based agentic AI applications; Automate infrastructure provisioning using Terraform, Bicep, or ARM templates.
Compensation: $100-$150/hour, please provide a specific hourly rate expectation Availability: 20-40 hours/week, with at least 6 hours of Pacific Time overlap/day Main tasks listed: Design, refine, and iterate on rubrics and evaluation criteria for preference data; Review and label data with a high quality bar, catching subtle issues others might overlook; Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows; Synthesize findings from data work into clear write-ups, updates, and recommendations for the team.
We are looking for experienced Data Analysts (MLE Bench) to contribute to benchmark-driven evaluation projects focused on real-world machine learning systems. This role involves hands-on analytical work with production-like datasets, metrics, and ML outputs to help evaluate, diagnose, and improve the performance of advanced AI systems.
We are looking for experienced Software Engineers (SWE Bench – Data Engineer / Data Science) to contribute to benchmark-driven evaluation projects focused on real-world data engineering and data science workflows. This role involves hands-on work with production-like datasets, data pipelines, and data science tasks to help evaluate and improve the performance of advanced AI systems.
About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc. Main tasks listed: Analyze and triage GitHub issues across trending open-source libraries; Set up and configure code repositories, including Dockerization and environment setup; Evaluating unit test coverage and quality; Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc. Main tasks listed: Analyze and triage GitHub issues across trending open-source libraries; Set up and configure code repositories, including Dockerization and environment setup; Evaluating unit test coverage and quality; Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.