AI Research Scientist: Scientific Evaluation (ML/AI)
Lightly AG · Remote · Worldwide
Salary: USD 50 – 65 an hour
Posted
A Zurich-based AI technology company is looking for experienced ML/AI researchers to evaluate research written by autonomous AI agents. This remote, flexible contract role pays $50–$65 per hour. It suits PhD-level researchers with at least three papers at NeurIPS, ICML, or ICLR who enjoy judging novelty, rigor, and scientific validity.
About the role
Introduction to the role
What happens when AI starts doing the research itself? Someone has to judge whether it is any good. A Zurich-based AI technology company is looking for experienced ML/AI researchers to take on that job. In this flexible, remote, project-based role, you will review papers written by autonomous AI research agents and decide whether the ideas are new, the experiments hold up, and the conclusions are earned. It is a rare chance to apply your research expertise to the evaluation of cutting-edge AI systems.
Who are we looking for?
You are an ML/AI researcher with a PhD in Machine Learning, AI, Computer Science, Statistics, or a closely related field, or equivalent demonstrated research expertise.
You are a published researcher with at least 3 peer-reviewed papers at NeurIPS, ICML, and/or ICLR, ideally including first-author or significant contributions.
You are hands-on, with experience developing novel methods, designing experiments, analysing results, and publishing original work.
You are a rigorous scientist with a strong grasp of baselines, statistical validity, benchmark design, ablation studies, and reproducibility.
You are an experienced reviewer, ideally for NeurIPS, ICML, ICLR, or similar conferences and journals (strongly preferred).
You are highly familiar with state-of-the-art ML/AI research and able to judge new methods against existing literature and benchmarks.You are comfortable verifying claims, from technical arguments and experimental evidence to citations and, where relevant, code.
You are a clear writer who can give concise, evidence-based feedback.
You are detail-oriented, quick to spot subtle flaws and gaps between claims and evidence.
You are disciplined, able to apply detailed scoring rubrics consistently while using independent expert judgment.
Your Responsibilities…
Research Evaluation
You will critically evaluate AI-generated ML/AI research papers, assessing their contributions, methodology, experimental design, and conclusions.
You will judge scientific novelty and significance, and whether the work advances meaningfully beyond the state of the art.
You will assess whether the research meets the standards of top-tier ML/AI conferences such as NeurIPS, ICML, and ICLR.
Rigor & Validity
You will examine experimental rigor, including baseline comparisons, benchmark choices, metrics, statistical evidence, and ablation studies.
You will identify methodological flaws, unsupported claims, inconsistencies, and misleading results.
You will assess reproducibility and technical soundness, checking whether findings are backed by evidence and, where available, code and implementation details.
Literature Verification
You will review relevant academic literature to find overlooked prior work, inaccurate citations, and unsupported novelty claims.
Structured Assessment
You will write structured, evidence-based assessments covering strengths, weaknesses, critical technical concerns, and overall scientific merit.
You will apply detailed evaluation rubrics consistently and justify each judgment clearly.
What Success looks like…
You will understand the role’s expectations, processes, tools, and performance standards.
You will carry out your responsibilities with confidence and increasing independence.
You will consistently deliver quality work and contribute effectively to team objectives.
You will demonstrate attention to detail, problem-solving ability, and effective communication.
You will follow established processes, adapt to feedback, and take ownership of your responsibilities.
To be considered for this role you should have…
A PhD in Machine Learning, AI, Computer Science, Statistics, or a closely related technical field, or equivalent demonstrated research expertise.
At least 3 peer-reviewed papers published at NeurIPS, ICML, and/or ICLR (a hard requirement).
Substantial hands-on ML/AI research experience.A strong understanding of scientific methodology and experimental rigor.
Experience reviewing papers for top-tier ML/AI conferences or journals (strongly preferred).
Strong analytical and scientific writing skills, and exceptional attention to detail.
Challenges you may face in this role...
Spotting subtle flaws and unsupported claims in polished, convincing AI-generated papers.
Judging genuine novelty against a fast-moving body of literature.
Applying detailed rubrics consistently while still using independent expert judgment.
Managing a flexible, project-based workload without direct oversight.
The Goodies…
Competitive hourly pay of $50–$65.
Fully remote work from anywhere, with flexible hours.
A part-time contract that fits around your other commitments.
The chance to work on the evaluation of frontier AI systems that conduct end-to-end scientific research.
A collaborative, inclusive environment where researchers of all backgrounds are welcomed and valued.
How to Apply
Interested candidates can apply here: https://join.com/companies/lightly/16880568/apply/authentication
Must-haves
- Machine Learning Research
- Scientific Peer Review
- Experimental Design & Statistical Validity
- Literature Review & Citation Verification
- Scientific Writing
Nice to have
- Conference Reviewing (NeurIPS / ICML / ICLR)
- Reproducibility & Code Review