Law Professors Prefer AI Answers in Stanford Study

A Stanford-led study found law professors preferred answers generated by large language models over those written by fellow instructors in roughly three out of four blinded comparisons.

Researchers recruited 16 professors from 14 U.S. law schools to create 40 contract law questions covering doctrine, case law, hypotheticals and policy. Faculty evaluators carried out 2,918 blinded head-to-head comparisons, choosing the answer they would rather give a student.

Google’s Gemini 2.5 Pro won 75.92% of its matchups and Google’s NotebookLM won 74.75%. In a broader analysis, Anthropic’s Claude Opus 4.7 and OpenAI’s ChatGPT 5.4 also ranked near the top. The study reported that every AI model evaluated outperformed human instructors on average.

The researchers examined model performance by question type and reported AI answers exceeded instructor answers on recall questions about cases, codes or doctrine, on hypotheticals, and on policy discussions. To test whether surface writing style explained preferences, they measured features such as answer length, organization, reasoning nuance, legal anchors, confidence tone, clarity and pedagogical support.

AI responses were flagged as harmful less often than professor-written answers. Gemini recorded a 3.41% harmfulness rate and NotebookLM 3.64%, compared with 12.06% for human instructors. The researchers also measured how often professors agreed when evaluating the same answer pairs and found observed agreement exceeded the level expected if judgments were entirely idiosyncratic.

The paper noted its design did not test whether AI responses matched each instructor’s individual teaching style. The authors wrote that their evaluation setting does not allow direct measurement of the extent to which instructor preferences are satisfied and that models may produce answers that are broadly acceptable rather than aligned with particular pedagogical choices.

The study appears alongside other developments in the legal field. The Los Angeles Superior Court is piloting an AI tool to summarize filings, organize evidence and generate draft rulings in civil cases. Law schools are adding AI training programs to prepare students for employer expectations. In April, a law firm informed a U.S. bankruptcy court that a filing contained AI-generated fabrications and distorted authorities, and the firm’s restructuring head wrote to the judge that the document included AI ‘hallucinations.’