ChatGPT Goes to Law School

16 Pages Posted: 25 Jan 2023 Last revised: 20 Oct 2023

See all articles by Jonathan H. Choi

Jonathan H. Choi

University of Southern California; University of Southern California Gould School of Law

Kristin E. Hickman

University of Minnesota - Twin Cities - School of Law

Amy Monahan

University of Minnesota Law School

Daniel Schwarcz

University of Minnesota Law School

Date Written: January 23, 2023

Abstract

How well can AI models write law school exams without human assistance? To find out, we used the widely publicized AI model ChatGPT to generate answers on four real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT performed on average at the level of a C+ student, achieving a low but passing grade in all four courses. After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.

Keywords: ChatGPT, law school, AI, natural language processing, Legal Data, NLP, Legal NLP, Legal Analytics, natural language understanding, evaluation, machine learning, artificial intelligence, artificial intelligence and law

Suggested Citation

Choi, Jonathan H. and Hickman, Kristin E. and Monahan, Amy and Schwarcz, Daniel, ChatGPT Goes to Law School (January 23, 2023). 71 Journal of Legal Education 387 (2022), Available at SSRN: https://ssrn.com/abstract=4335905 or http://dx.doi.org/10.2139/ssrn.4335905

Jonathan H. Choi (Contact Author)

University of Southern California ( email )

2250 Alcazar Street
Los Angeles, CA 90089
United States

University of Southern California Gould School of Law ( email )

699 Exposition Blvd.
Los Angeles, CA 90089
United States

Kristin E. Hickman

University of Minnesota - Twin Cities - School of Law ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States
612-624-2915 (Phone)

Amy Monahan

University of Minnesota Law School ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States

Daniel Schwarcz

University of Minnesota Law School ( email )

229 19th Avenue South
Minneapolis, MN 55455
United States

HOME PAGE: http://www.law.umn.edu/profiles/daniel-schwarcz

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
14,890
Abstract Views
56,014
Rank
516
PlumX Metrics