AI Outperforms Law Professors in Stanford Study
A Stanford Law School study reveals AI-generated answers are preferred by law professors over peer-written responses, challenging assumptions about AI's role in legal education
Recent research from Stanford Law School has found that law professors prefer AI-generated answers to student questions over responses written by their fellow instructors. This groundbreaking study, led by Professor Julian Nyarko, has significant implications for the future of legal education. The study's findings suggest that AI can serve as an effective tool for supporting student learning, particularly in subjects that require nuanced reasoning and judgment.
What happened
The study, titled "Law Professors Prefer AI Over Peer Answers," was conducted with 16 law professors across U.S. law schools and tested whether large language models could serve as effective tutors for contract law courses. In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors. The study found that AI systems performed comparably to the best human instructor in the study, with professors flagging AI responses as pedagogically harmful only 3.5% of the time, compared to 12% for peer-written answers.
The study's methodology involved having participants create 40 representative contracts law questions that students might ask after class or during office hours, write their own answers, and then evaluate responses without knowing whether they came from AI or other participating professors. The AI systems were calibrated to match the length and structure of human answers, and multiple evaluation methods were used to ensure the study's validity.
The research team took extensive precautions to ensure the study's validity, including calibrating AI responses to match the length and structure of human answers, using multiple evaluation methods, and having professors assess whether responses might mislead or confuse students. The study also examined specific AI models, including commercial tutoring systems and Google's NotebookLM, finding varying levels of performance.
Why it matters
The study's findings have significant implications for the future of legal education. The fact that AI-generated answers were preferred by law professors over peer-written responses suggests that AI can serve as an effective tool for supporting student learning, particularly in subjects that require nuanced reasoning and judgment. This challenges important assumptions about AI's role in legal education and raises questions about how AI can best be used to support student learning.
The study's findings also have implications for the way that legal education is delivered. If AI can provide high-quality, on-demand support that complements classroom instruction, it could potentially broaden access to expert guidance and improve student outcomes. However, the study's authors are emphatic that quality and deployment are separate questions, and they have only addressed the first.
- AI-generated answers can provide high-quality, on-demand support that complements classroom instruction
- AI can potentially broaden access to expert guidance and improve student outcomes
- AI can serve as an effective tool for supporting student learning, particularly in subjects that require nuanced reasoning and judgment
- There is a risk that students may become over-reliant on AI and neglect to develop their own critical thinking skills
- There are concerns about the potential for AI to hallucinate or provide misleading information
- There are questions about how AI can be effectively integrated into legal education while maintaining rigorous academic standards
How to think about it
When considering the use of AI in legal education, it is essential to think critically about the potential benefits and drawbacks. While AI can provide high-quality, on-demand support that complements classroom instruction, it is crucial to ensure that students are not becoming over-reliant on AI and are still developing their own critical thinking skills. It is also important to consider the potential risks associated with AI, such as the potential for hallucination or misleading information.