At Associated Press, Garance Burke and Hilke Schellmann report that OpenAI claims that its transcription tool Whisper has near “human level robustness and accuracy.” Yet the firm also warns against using the tools in “high-risk domains.”

Further:

… Whisper has a major flaw: It is prone to making up chunks of text or even entire sentences, according to interviews with more than a dozen software engineers, developers and academic researchers. Those experts said some of the invented text — known in the industry as hallucinations — can include racial commentary, violent rhetoric and even imagined medical treatments.

“Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said,” October 25, 2024

One researcher, they say, found hallucinations in eight out of 10 AI transcriptions of public meetings. Others note similar findings.

Non-existent medicines

The problems this could create in health care are obvious. For example, a snippet from the research: “In a third transcription, Whisper invented a non-existent medication called ‘hyperactivated antibiotics.’” That’s why OpenAI has warned against using its tool in “decision-making contexts.” But many appear to have started doing so anyway.

Burke and Schellman report that OpenAI is working to address the problem. The difficulty is that a tendency to hallucinate may be baked into the way its AI models operate. In the case of chatbots, for example, continuing to reprocess their own data leads to hallucination and model collapse.

A recent report by six AI researchers reported recently that large language models (chatbots) just don’t do formal reasoning. Perhaps some kinds of thinking just can’t be mechanized.

You may also wish to read: Philosopher: Human language is too entangled for computers. Noë takes issue with the approach advocated by Alan Turing where we can follow certain steps to have a thinking computer. He doubts such steps are possible.