BBC Discovers Limitations of AI Chatbots in Summarizing News
A BBC research has found that some AI chatbots are unable to accurately summarise news.
The news is a significant part of life, telling stories from around the world, and with the influx of AI chatbots, things should be easier. However, that's not often the case, especially as research carried out by the BBC reveals that four leading artificial intelligence (AI) chatbots are inaccurately summarising news stories.
Risky Innovations
According to the report, the BBC gave OpenAI's ChatGPT, Microsoft's Copilot, Google's Gemini and Perplexity AI content from the BBC website, and asked them questions about the news. The resulting answers were bogged by "significant inaccuracies" and distortions. Deborah Turness, the CEO of BBC News and Current Affairs, said AI has come with "endless opportunities" but the companies developing the tools were "playing with fire".
"We live in troubled times, and how long will it be before an AI-distorted headline causes significant real-world harm?", she asked. "We support publishers and creators by helping 300 million weekly ChatGPT users discover quality content through summaries, quotes, clear links, and attribution," OpenAI responded via a spokesperson.
Questionable Reporting
In the study, the BBC asked Gemini, ChatGPT, Copilot, and Perplexity to summarise 100 news stories and rate each answer. Journalists relevant to the subject of the article were also asked to rate the quality of answers from the AI assistants. The study found that 51% of all AI answers to questions about the news were judged to have some form of significant issue.
Further, 19% of AI answers that cited BBC content introduced factual errors, including incorrect factual statements, numbers, and dates. On her part, Turness said the BBC was seeking to "open up a new conversation with AI tech providers" so we can "work together in partnership to find solutions". She further called on the tech companies to "pull back" their AI news summaries, citing how Apple pulled its news stories after complaints from the BBC that Apple Intelligence was misrepresenting news stories.
Unveiled Inaccuracies
Some of the inaccuracies found by the BBC study include Gemini incorrectly stating that the NHS did not recommend vaping as an aid to quitting smoking, while ChatGPT and Copilot said Rishi Sunak and Nicola Sturgeon were still in office, long after they had left. Perplexity also misquoted a BBC news piece on the Middle East, stating Iran initially showed "restraint" and described Israel's actions as "aggressive."
The study noted that Microsoft's Copilot and Google's Gemini had more significant issues than Perplexity and OpenAI's ChatGPT. Before now, the BBC blocked its content from AI chatbots, but it opened up its website for the duration of the tests in December 2024.
The report further revealed that apart from factual inaccuracies, the chatbots "struggled to differentiate between opinion and fact, editorialized, and often failed to include essential context".
Pete Archer, BBC's Programme Director for Generative AI, said publishers "should have control over whether and how their content is used and AI companies should show how assistants process news along with the scale and scope of errors and inaccuracies they produce."
As AI technology progresses, there's a need to maintain truth and accuracy, particularly with news headlines. Misleading pieces and inaccuracies could lead to damaging consequences like conflicts between countries and human rights violations.
