Senior Software Engineer – AI Evaluation

Senior Software Engineer – AI Evaluation

About the Role

What if your engineering skills could directly determine whether the world's most advanced AI systems are actually working? We're looking for Senior Software Engineers to design, build, and scale the evaluation infrastructure that measures AI performance — the critical layer between raw model output and real-world trust.

This is high-impact, technically challenging work at the intersection of software engineering and AI research. You'll build the tools, pipelines, and frameworks that help leading research teams understand what their models can do, where they fail, and how to make them better. If you love building robust systems and care deeply about quality and measurement, this role puts you at the center of the AI revolution.

• Organization: Alignerr
• Type: Hourly Contract
• Location: Remote
• Commitment: 20–40 hours/week

What You'll Do

• Design and build scalable evaluation pipelines and frameworks for assessing AI model performance across diverse tasks and domains
• Develop automated testing harnesses, scoring systems, and benchmarking tools for large language models and other AI systems
• Write clean, production-quality code to process, analyze, and visualize evaluation datasets at scale
• Create and maintain APIs, dashboards, and internal tools that enable research teams to run, track, and compare evaluations efficiently
• Collaborate with AI researchers and data scientists to translate evaluation methodologies into reliable, repeatable software
• Identify edge cases, failure modes, and reliability issues in AI outputs through systematic engineering approaches
• Optimize system performance, data processing speed, and infrastructure costs
• Contribute to the architecture and technical direction of the evaluation platform
• Write clear documentation and participate in code reviews to maintain high engineering standards

Who You Are

• 5+ years of professional software engineering experience, with a track record of building and shipping production systems
• Strong proficiency in Python — including experience with data processing libraries (pandas, NumPy) and web frameworks (FastAPI, Flask, or Django)
• Solid understanding of software architecture, design patterns, and engineering best practices
• Experience working with large datasets and building data pipelines
• Comfortable with cloud infrastructure (AWS, GCP, or Azure) and containerized deployments
• Familiarity with version control (Git), CI/CD workflows, and testing frameworks
• Strong problem-solving skills and the ability to work through ambiguity independently
• Excellent written communication skills — you can document your work clearly and collaborate asynchronously
• Self-motivated and reliable when working independently in a remote environment

Nice to Have

• Experience with ML/AI evaluation, benchmarking, or model testing
• Familiarity with LLMs, prompt engineering, or AI safety and alignment concepts
• Background in building developer tools, internal platforms, or data infrastructure
• Experience with distributed systems, message queues, or workflow orchestration (Airflow, Prefect, etc.)
• Knowledge of statistical methods for measuring and comparing model performance
• Prior experience in a remote-first or async-first engineering culture
• Contributions to open-source projects related to AI, ML, or evaluation tooling

Why Join Us

• Work on cutting-edge AI evaluation projects alongside world-class research labs
• Directly influence how AI quality and safety are measured at scale — your code shapes the standard
• Fully remote and flexible — work when and where you're most productive
• Freelance autonomy with access to deeply meaningful, technically stimulating work
• Collaborate with a global team of engineers and researchers pushing the boundaries of AI
• Exposure to the latest developments in AI research, model capabilities, and evaluation science
• Potential for ongoing work and contract extension as the platform and project scope grow