In a striking development underscoring the growing pains of artificial intelligence in professional services, Sullivan & Cromwell, one of New York City’s most prestigious law firms, has publicly apologized for submitting a court document containing fabricated citations generated by AI software. The firm acknowledged that the so-called AI “hallucinations” led to errors in the filing, raising fresh questions about the reliability and oversight of AI tools in high-stakes legal environments.
The incident occurred amid a broader surge of AI adoption in legal practices across New York, where firms are increasingly experimenting with generative AI to draft documents, conduct case research, and streamline workflows. However, this episode at Sullivan & Cromwell serves as a cautionary tale about the technology’s limitations. “Hallucinations” refer to AI’s tendency to produce plausible but entirely fictitious information — in this case, fake legal citations that do not exist in any court record.
Industry insiders suggest that while AI can enhance efficiency, the responsibility remains with human attorneys to rigorously verify all outputs before submission to the court. Sullivan & Cromwell’s swift acknowledgment and apology reflect the firm’s commitment to maintaining its rigorous standards, yet the misstep has sparked debate over how law firms in New York should integrate AI to avoid similar pitfalls.
This incident also highlights the unique pressures faced by New York City’s legal sector, where clients demand rapid turnaround times and flawless accuracy. As AI continues to reshape the city’s legal landscape, firms must balance innovation with caution. Experts predict that regulatory scrutiny and updated ethical guidelines for AI usage in law will likely follow.
For now, Sullivan & Cromwell’s experience serves as a high-profile example of the challenges New York’s legal community faces as it navigates the uncharted waters of artificial intelligence — a reminder that even the most elite institutions are not immune to the technology’s growing pains.