Beyond the Mirage: Beware of Generative AI and Hallucinations; New York State Bar Association (NYSBA), June 26, 2026

 Cynthia Feathers , New York State Bar Association (NYSBA); Beyond the Mirage: Beware of Generative AI and Hallucinations

"The work of attorneys can be arduous. With demanding caseloads comes an openness to tools that can help us do our jobs more efficiently. The advent of online legal research was a game changer for attorneys decades ago. In recent years, generative artificial intelligence – AI that can create original content such as text, images, video, audio or software code in response to a user’s prompt or request – has begun to revolutionize legal practice.[1] This development has included the integration of AI into legal research and writing.

The focus here is on the risks inherent in popular generative AI models used to complete such tasks: They are prone to producing false legal information, so-called hallucinations, including false case citations and false reasoning, quotes and holdings. A nationwide epidemic of cases involving such fabrications has made the risks of unverified AI use well known. Hundreds of decisions have touched on this issue.[2] Thus, the legal profession has been alerted that blind faith in generative AI results is misplaced.

This article makes no attempt to be exhaustive as to the rapidly unfolding case law but does seek to highlight some emblematic decisions issued by state, federal, trial and appellate courts throughout the country over the last two years and to bring attention to the dangers of failing to check AI results...

Evolving technology is seductive in creating the illusion that it can save us from the hard work. But our ethical duties to our clients and the courts still require that we rigorously verify every case cited. When generative AI output becomes more reliable, new questions will arise about how far we can go in abdicating our lawyerly judgment to new technology.[20]

For now, New York attorneys should be aware of a new rule on AI adopted by the New York State Unified Court System. Effective June 1, Part 161 of the Rules of the Chief Administrator of the Courts permits the use of AI tools in preparing submissions to a court and does not require the disclosure of such use. However, Part 161 sets forth a model rule that does require attorneys using such tools to “carefully review” each submission and “independently ensure” that they do not contain “fabricated or fictitious cases, statutes, or other material.”[21]  Individual judges retain discretion to implement their own AI-related rules, adopt the model rule or impose no additional requirements through their part rules.

Perhaps soon we will see more standing orders on AI use and updated ethics rules nationwide targeting AI issues.[22]  In the meantime, in the use of AI, we can be guided by the new court rule and longstanding mandates regarding competence, diligence, accuracy and candor and the supervision of lawyers and nonlawyers."