Showing posts with label AI hallucinations. Show all posts
Showing posts with label AI hallucinations. Show all posts

Friday, July 25, 2025

Virginia teachers learn AI tools and ethics at largest statewide workshop; WTVR, July 23, 2025

 

Wednesday, July 23, 2025

Partner Who Wrote About AI Ethics, Fired For Citing Fake AI Cases; Above The Law, July 23, 2025

Joe Patrice , Above The Law; Partner Who Wrote About AI Ethics, Fired For Citing Fake AI Cases

"Don’t blame the AI for the fact that you read a brief and never bothered to print out the cases. Who does that? Long before AI, we all understood that you needed to look at the case itself to make sure no one missed the literal red flag on top. It might’ve ended up in there because of AI, but three lawyers and presumably a para or two had this brief and no one built a binder of the cases cited? What if the court wanted oral argument? No one is excusing the decision to ask ChatGPT to resolve your $24 million case, but the blame goes far deeper.

Malaty will shoulder most of the blame as the link in the workflow who should’ve known better. That said, her article about AI ethics, written last year, doesn’t actually address the hallucination problem. While risks of job displacement and algorithms reinforcing implicit bias are important, it is a little odd to write a whole piece on the ethics of legal AI without even breathing on hallucinations."

Tuesday, July 22, 2025

Getting Along with GPT: The Psychology, Character, and Ethics of Your Newest Professional Colleague; ABA Journal, May 9, 2025

 ABA Journal; Getting Along with GPT: The Psychology, Character, and Ethics of Your Newest Professional Colleague

"The Limits of GenAI’s Simulated Humanity

  • Creative thinking. An LLM mirrors humanity’s collective intelligence, shaped by everything it has read. It excels at brainstorming and summarizing legal principles but lacks independent thought, opinions, or strategic foresight—all essential to legal practice. Therefore, if a model’s summary of your legal argument feels stale, illogical, or disconnected from human values, it may be because the model has no democratized data to pattern itself on. The good news? You may be on to something original—and truly meaningful!
  • True comprehension. An LLM does not know the law; it merely predicts legal-sounding text based on past examples and mathematical probabilities.
  • Judgment and ethics. An LLM does not possess a moral compass or the ability to make judgments in complex legal contexts. It handles facts, not subjective opinions.  
  • Long-term consistency. Due to its context window limitations, an LLM may contradict itself if key details fall outside its processing scope. It lacks persistent memory storage.
  • Limited context recognition. An LLM has limited ability to understand context beyond provided information and is limited by training data scope.
  • Trustfulness. Attorneys have a professional duty to protect client confidences, but privacy and PII (personally identifiable information) are evolving concepts within AI. Unlike humans, models can infer private information without PII, through abstract patterns in data. To safeguard client information, carefully review (or summarize with AI) your LLM’s terms of use."

Wednesday, July 16, 2025

The Pentagon is throwing $200 million at ‘Grok for Government’ and other AI companies; Task & Purpose, July 14, 2025

 , Task & Purpose; The Pentagon is throwing $200 million at ‘Grok for Government’ and other AI companies

"The Pentagon announced Monday it is going to spend almost $1 billion on “agentic AI workflows” from four “frontier AI” companies, including Elon Musk’s xAI, whose flagship Grok appeared to still be declaring itself “MechaHitler” as late as Monday afternoon.

In a press release, the Defense Department’s Chief Digital and Artificial Intelligence Office — or CDAO — said it will cut checks of up to $200 million each to tech giants Anthropic, Google, OpenAI and Musk’s xAI to work on:

  • “critical national security challenges;”
  • “joint mission essential tasks in our warfighting domain;”
  • “DoD use cases.”

The release did not expand on what any of that means or how AI might help. Task & Purpose reached out to the Pentagon for details on what these AI agents may soon be doing and asked specifically if the contracts would include control of live weapons systems or classified information."

Wednesday, July 2, 2025

Trial Court Decides Case Based On AI-Hallucinated Caselaw; Above The Law, July 1, 2025

Joe Patrice, Above The Law; Trial Court Decides Case Based On AI-Hallucinated Caselaw

"Between opposing counsel and diligent judges, fake cases keep getting caught before they result in real mischief. That said, it was always only a matter of time before a poor litigant representing themselves fails to know enough to sniff out and flag Beavis v. Butthead and a busy or apathetic judge rubberstamps one side’s proposed order without probing the cites for verification. Hallucinations are all fun and games until they work their way into the orders.

It finally happened with a trial judge issuing an order based off fake cases (flagged by Rob Freund(Opens in a new window)). While the appellate court put a stop to the matter, the fact that it got this far should terrify everyone.

Shahid v. Esaam(Opens in a new window), out of the Georgia Court of Appeals, involved a final judgment and decree of divorce served by publication. When the wife objected to the judgment based on improper service, the husband’s brief included two fake cases. The trial judge accepted the husband’s argument, issuing an order based in part on the fake cases."

Saturday, June 21, 2025

US patent office wants an AI to scan for prior art, but doesn't want to pay for it; The Register, June 20, 2025

Brandon Vigliarolo,  The Register; US patent office wants an AI to scan for prior art, but doesn't want to pay for it

"There is some irony in using AI bots, which are often trained on copyrighted material for which AI firms have shown little regard, to assess the validity of new patents. 

It may not be the panacea the USPTO is hoping for. Lawyers have been embracing AI for something very similar - scanning particular, formal documentation for specific details related to a new analysis - and it's sometimes backfired as the AI has gotten certain details wrong. The Register has reported on numerous instances of legal professionals practically begging to be sanctioned for not bothering to do their legwork, as judges caught them using AI, which borked citations to other legal cases. 

The risk of hallucinating patents that don't exist, or getting patent numbers or other details wrong, means that there'll have to be at least some human oversight. The USPTO had no comment on how this might be accomplished."

Monday, June 2, 2025

Excruciating reason Utah lawyer presented FAKE case in court after idiotic blunder; Daily Mail, May 31, 2025

JOE HUTCHISON FOR DAILYMAIL.COMExcruciating reason Utah lawyer presented FAKE case in court after idiotic blunder

"The case referenced, according to documents, was 'Royer v. Nelson' which did not exist in any legal database and was found to be made up by ChatGPT.

Opposing counsel said that the only way they would find any mention of the case was by using the AI

They even went as far as to ask the AI if the case was real, noting in a filing that it then apologized and said it was a mistake.

Bednar's attorney, Matthew Barneck, said that the research was done by a clerk and Bednar took all responsibility for failing to review the cases.

He told The Salt Lake Tribune: 'That was his mistake. He owned up to it and authorized me to say that and fell on the sword."

Friday, May 30, 2025

White House MAHA Report may have garbled science by using AI, experts say; The Washington Post, May 29, 2025

, The Washington Post ; White House MAHA Report may have garbled science by using AI, experts say

"Some of the citations that underpin the science in the White House’s sweeping “MAHA Report” appear to have been generated using artificial intelligence, resulting in numerous garbled scientific references and invented studies, AI experts said Thursday.

Of the 522 footnotes to scientific research in an initial version of the report sent to The Washington Post, at least 37 appear multiple times, according to a review of the report by The Post. Other citations include the wrong author, and several studies cited by the extensive health report do not exist at all, a fact first reported by the online news outlet NOTUS on Thursday morning.

Some references include “oaicite” attached to URLs — a definitive sign that the research was collected using artificial intelligence. The presence of “oaicite” is a marker indicating use of OpenAI, a U.S. artificial intelligence company."

Wednesday, May 21, 2025

A.I.-Generated Reading List in Chicago Sun-Times Recommends Nonexistent Books; The New York Times, May 21, 2025

 , The New York Times; A.I.-Generated Reading List in Chicago Sun-Times Recommends Nonexistent Books

"The summer reading list tucked into a special section of The Chicago Sun-Times and The Philadelphia Inquirer seemed innocuous enough.

There were books by beloved authors such as Isabel Allende and Min Jin Lee; novels by best sellers including Delia Owens, Taylor Jenkins Reid and Brit Bennett; and a novel by Percival Everett, a recent Pulitzer Prize winner.

There was just one issue: None of the book titles attributed to the above authors were real. They had been created by generative artificial intelligence.

It’s the latest case of bad A.I. making its way into the news. While generative A.I. has improved, there is still no way to ensure the systems produce accurate information. A.I. chatbots cannot distinguish between what is true and what is false, and they often make things up. The chatbots can spit out information and expert names with an air of authority."

Saturday, May 17, 2025

Anthropic’s law firm throws Claude under the bus over citation errors in court filing; The Register, May 15, 2025

Thomas Claburn, The Register; Anthropic’s law firm throws Claude under the bus over citation errors in court filing

"An attorney defending AI firm Anthropic in a copyright case brought by music publishers apologized to the court on Thursday for citation errors that slipped into a filing after using the biz's own AI tool, Claude, to format references.

The incident reinforces what's becoming a pattern in legal tech: while AI models can be fine-tuned, people keep failing to verify the chatbot's output, despite the consequences.

The flawed citations, or "hallucinations," appeared in an April 30, 2025 declaration [PDF] from Anthropic data scientist Olivia Chen in a copyright lawsuit music publishers filed in October 2023.

But Chen was not responsible for introducing the errors, which appeared in footnotes 2 and 3.

Ivana Dukanovic, an attorney with Latham & Watkins, the firm defending Anthropic, stated that after a colleague located a supporting source for Chen's testimony via Google search, she used Anthropic's Claude model to generate a formatted legal citation. Chen and defense lawyers failed to catch the errors in subsequent proofreading.

"After the Latham & Watkins team identified the source as potential additional support for Ms. Chen’s testimony, I asked Claude.ai to provide a properly formatted legal citation for that source using the link to the correct article," explained Dukanovic in her May 15, 2025 declaration [PDF].

"Unfortunately, although providing the correct publication title, publication year, and link to the provided source, the returned citation included an inaccurate title and incorrect authors.

"Our manual citation check did not catch that error. Our citation check also missed additional wording errors introduced in the citations during the formatting process using Claude.ai."...

The hallucinations of AI models keep showing up in court filings.

Last week, in a plaintiff's claim against insurance firm State Farm (Jacquelyn Jackie Lacey v. State Farm General Insurance Company et al), former Judge Michael R. Wilner, the Special Master appointed to handle the dispute, sanctioned [PDF] the plaintiff's attorneys for misleading him with AI-generated text. He directed the plaintiff's legal team to pay more than $30,000 in court costs that they wouldn't have otherwise had to bear.

After reviewing a supplemental brief filed by the plaintiffs, Wilner found that "approximately nine of the 27 legal citations in the ten-page brief were incorrect in some way."

Two of the citations, he said, do not exist, and several cited phony judicial opinions."

Thursday, May 15, 2025

Anthropic expert accused of using AI-fabricated source in copyright case; Reuters, May 13, 2025

 , Reuters; Anthropic expert accused of using AI-fabricated source in copyright case

"Van Keulen asked Anthropic to respond by Thursday to the accusation, which the company said appeared to be an inadvertent citation error. He rejected the music companies' request to immediately question the expert but said the allegation presented "a very serious and grave issue," and that there was "a world of difference between a missed citation and a hallucination generated by AI.""

Wednesday, May 14, 2025

The Professors Are Using ChatGPT, and Some Students Aren’t Happy About It; The New York Times, May 14, 2025

 , The New York Times; The Professors Are Using ChatGPT, and Some Students Aren’t Happy About It

"When ChatGPT was released at the end of 2022, it caused a panic at all levels of education because it made cheating incredibly easy. Students who were asked to write a history paper or literary analysis could have the tool do it in mere seconds. Some schools banned it while others deployed A.I. detection services, despite concerns about their accuracy.

But, oh, how the tables have turned. Now students are complaining on sites like Rate My Professors about their instructors’ overreliance on A.I. and scrutinizing course materials for words ChatGPT tends to overuse, like “crucial” and “delve.” In addition to calling out hypocrisy, they make a financial argument: They are paying, often quite a lot, to be taught by humans, not an algorithm that they, too, could consult for free."

Friday, March 7, 2025

AI 'hallucinations' in court papers spell trouble for lawyers; Reuters, February 18, 2025

, Reuters ; AI 'hallucinations' in court papers spell trouble for lawyers

"U.S. personal injury law firm Morgan & Morgan sent an urgent email this month to its more than 1,000 lawyers: Artificial intelligence can invent fake case law, and using made-up information in a court filing could get you fired.

A federal judge in Wyoming had just threatened to sanction two lawyers at the firm who included fictitious case citations in a lawsuit against Walmart. One of the lawyers admitted in court filings last week that he used an AI program that "hallucinated" the cases and apologized for what he called an inadvertent mistake."

Monday, February 3, 2025

DeepSeek has ripped away AI’s veil of mystique. That’s the real reason the tech bros fear it; The Observer via The Guardian, February 2, 2025

, The Observer via The Guardian ; DeepSeek has ripped away AI’s veil of mystique. That’s the real reason the tech bros fear it

"DeepSeek, sponsored by a Chinese hedge fund, is a notable achievement. Technically, though, it is no advance on large language models (LLMs) that already exist. It is neither faster nor “cleverer” than OpenAI’s ChatGPT or Anthropic’s Claude and just as prone to “hallucinations” – the tendency, exhibited by all LLMs, to give false answers or to make up “facts” to fill gaps in its data. According to NewsGuard, a rating system for news and information websites, DeepSeek’s chatbot made false claims 30% of the time and gave no answers to 53% of questions, compared with 40% and 22% respectively for the 10 leading chatbots in NewsGuard’s most recent audit.

The figures expose the profound unreliability of all LLMs. DeepSeek’s particularly high non-response rate is likely to be the product of its censoriousness; it refuses to provide answers on any issue that China finds sensitive or about which it wants facts restricted, whether Tiananmen Square or Taiwan...

Nevertheless, for all the pushback, each time one fantasy prediction fails to materialise, another takes its place. Such claims derive less from technological possibilities than from political and economic needs. While AI technology has provided hugely important tools, capable of surpassing humans in specific fields, from the solving of mathematical problems to the recognition of disease patterns, the business model depends on hype. It is the hype that drives the billion-dollar investment and buys political influence, including a seat at the presidential inauguration."

Monday, January 27, 2025

Beyond ChatGPT: WVU researchers to study use and ethics of artificial intelligence across disciplines; WVU Today, January 22, 2025

WVU Today; Beyond ChatGPT: WVU researchers to study use and ethics of artificial intelligence across disciplines

"Two West Virginia University researchers have designed a curriculum to engage liberal arts faculty in discussions on the social, ethical and technical aspects of artificial intelligence and its role in classrooms.

Through a grant from the National Endowment for the Humanities, Erin Brock Carlson, assistant professor of English, and Scott Davidson, professor of philosophy, both at the WVU Eberly College of Arts and Sciences, have designed an interdisciplinary, cross-institutional program to facilitate conversations among faculty about the benefits and drawbacks of AI, how it functions and the need for human interpretation.

The award will fund a summer workshop in which Carlson and Davidson will offer AI trainings for humanities faculty and guide them through creation and development of courses with an AI component. The researchers will then assist as faculty offer those courses to students, assess progress and help with the implementation of the projects that develop.

The researchers said they hope to challenge the notion that artificial intelligence research falls into the domain of STEM fields. 

“The humanities gets overlooked and underappreciated so often,” Carlson said. “We are doing important, meaningful research, just like our colleagues in STEM and other fields. This is a chance to use a humanities lens to examine contemporary problems and developments like artificial intelligence and also to get conversations going between fields that oftentimes don’t talk to one another as much as we should.”

Co-directors Carlson and Davidson will be joined by a team of mentors and fellows — two from data science fields and two from the humanities perspective — that will serve and assist as resources in the interdisciplinary conversations. The seminar and summer workshops will support the creation or redesign of 10 courses. They plan to invite off-campus experts to help facilitate the workshops, work with the faculty and support their projects.

“It’s really about expanding capacity at the University and in the humanities to investigate the implications of AI or to actually use AI in humanities courses, whether it’s for writing, creating art or creating projects through the use of AI,” Davidson said. “There are a lot of different possibilities and directions that we hope these courses take. If we have 10 of them, it’s really going to have a big impact on humanities education here at the University.”

Carlson and Davidson acknowledge that attitudes about AI tend to be either extremely optimistic or extremely skeptical but that the reality is somewhere in the middle.

“AI is such a simplistic term to describe a whole suite of different technologies and developments that folks are dealing with every day, whether they know it or not,” Carlson said, noting that discussions could focus on personal, social and economic impacts of AI use, as well as how it affects character and intellectual values. 

Davidson was inspired to focus on AI when he found an erroneous, AI-generated summary of one of his own articles.

“It was totally wrong,” he said. “I didn’t say those things, and it made me think about how somebody might look me up and find that summary of my article and get this false impression of me. That really highlighted that we need to build an understanding in students of the need to inquire deeper and to understand that you have to be able to evaluate AI’s accuracy and its reliability.”

Carlson and Davidson said the conversations need to consider AI’s drawbacks, as well. Using AI consumes large amounts of water and electricity resulting in greenhouse emissions. Data centers produce electronic waste that can contain mercury and lead. 

They also intend to follow legal cases and precedents surrounding the use of AI.

“That’s another aspect of AI and the ways that it represents people,” Carlson said. “Because it has a very real, material impact on people in communities. It’s not just a super computer in a room. It’s a network that has a bunch of different implications for a bunch of different people, ranging from jobs to familial relationships. That’s the value of the humanities — to ask these tough questions because it’s increasingly difficult to avoid all of it.”

Conversations, as they expand, will need to keep up with the pace of AI’s rapidly developing landscape.  

“There’s going to be a lot of people involved in this,” she said. “We put together an amazing team. We want it to be an open, honest and ethical conversation that brings in other folks and opens up further conversations across the College and the University at large.”"

Friday, January 17, 2025

Apple sidelines AI news summaries due to errors; Japan Today, January 17, 2025

Japan Today; Apple sidelines AI news summaries due to errors

"Apple pushed out a software update on Thursday which disabled news headlines and summaries generated using artificial intelligence that were lambasted for getting facts wrong.

The move by the tech titan comes as it enhances its latest lineup of devices with "Apple Intelligence" in a market keen for assurance that the iPhone maker is a contender in the AI race.

Apple's decision to temporarily disable the recently launched AI feature comes after the BBC and other news organizations complained that users were getting mistake-riddled or outright wrong headlines or news summary alerts."

Friday, December 27, 2024

The Job Interview Question Everyone Will Be Asking In 2025; Forbes, December 26, 2024

 Chris Westfall, Forbes; 

The Job Interview Question Everyone Will Be Asking In 2025

"Inside job interview questions, a new number one topic has emerged. Beyond the usual inquiries around your background and experience, the theme that’s top of mind is artificial intelligence (AI). The number one question every candidate should anticipate in 2025 is this one: How familiar are you with AI, and how are you using it? Here’s how to prepare, and respond, to the new number one job interview question.

As with any job interview question, the best answer usually involves a story. Because the minute you say, “I’m very familiar with AI,” the interviewer would like you to prove it. You can say you’re a genius, super empathetic, trustworthy, or the world’s fastest coder - the tricky part is providing credible evidence. Saying you are familiar with something is not the same as demonstrating it. That’s where soft skills like communication come into play."

Thursday, December 26, 2024

How Hallucinatory A.I. Helps Science Dream Up Big Breakthroughs; The New York Times, December 23, 2024

, The New York Times; How Hallucinatory A.I. Helps Science Dream Up Big Breakthroughs

"In the universe of science, however, innovators are finding that A.I. hallucinations can be remarkably useful. The smart machines, it turns out, are dreaming up riots of unrealities that help scientists track cancer, design drugs, invent medical devices, uncover weather phenomena and even win the Nobel Prize.

“The public thinks it’s all bad,” said Amy McGovern, a computer scientist who directs a federal A.I. institute. “But it’s actually giving scientists new ideas. It’s giving them the chance to explore ideas they might not have thought about otherwise.”

The public image of science is coolly analytic. Less visibly, the early stages of discovery can teem with hunches and wild guesswork. “Anything goes” is how Paul Feyerabend, a philosopher of science, once characterized the free-for-all.

Now, A.I. hallucinations are reinvigorating the creative side of science. They speed the process by which scientists and inventors dream up new ideas and test them to see if reality concurs. It’s the scientific method — only supercharged. What once took years can now be done in days, hours and minutes. In some cases, the accelerated cycles of inquiry help scientists open new frontiers."

Wednesday, December 25, 2024

Should you trust an AI-assisted doctor? I visited one to see.; The Washington Post, December 25, 2024

, The Washington Post; Should you trust an AI-assisted doctor? I visited one to see.

"The harm of generative AI — notorious for “hallucinations” — producing bad information is often difficult to see, but in medicine the danger is stark. One study found that out of 382 test medical questions, ChatGPT gave an “inappropriate” answer on 20 percent. A doctor using the AI to draft communications could inadvertently pass along bad advice.

Another study found that chatbots can echo doctors’ own biases, such as the racist assumption that Black people can tolerate more pain than White people. Transcription software, too, has been shown to invent things that no one ever said."

Tuesday, December 10, 2024

OpenAI makes AI video generator Sora publicly available in US; The Guardian, December 9, 2024

 , The Guardian; OpenAI makes AI video generator Sora publicly available in US

"Anyone in the US can now use OpenAI’s artificial intelligence video generator, Sora, which the company announced on Monday would become publicly available. OpenAI first presented Sora in February, but it was only accessible to select artists, film-makers and safety testers. At multiple points on Monday, though, OpenAI’s website did not allow for new sign-ups for Sora, citing heavy traffic...

While generative AI has improved considerably over the past year, it is still prone to hallucinations, or incorrect responses, and plagiarism. AI image generators also often produce unrealistic images, such as people with several arms or misplaced facial features.

Critics warn that this type of AI video technology could be misused by bad actors for disinformation, scams and deepfakes. There have already been deepfake videos of the Ukrainian president, Volodymyr Zelenskyy, supposedly calling for a ceasefire and of Kamala Harris supposedly describing herself as “the ultimate diversity hire”.

OpenAI said in its blogpost that it would initially limit uploads of specific people and that it will block content with nudity. The company said that it was additionally “blocking particularly damaging forms of abuse, such as child sexual abuse materials and sexual deepfakes”.

Sora will be available to users who already subscribe and pay for OpenAI’s tools. People in the US and “most countries internationally” will have access to the tool, but it will not be available in the UK or Europe."