ChatGPT Made BCG Consultants Look Like Data Scientists Until It Was Taken Away, Nature Study Finds
A newly peer-reviewed trial of 487 BCG consultants, run in 2024 on GPT-4-era ChatGPT, found the chatbot lifted their data science work but left no measurable new knowledge once it was gone. The study was co-authored by BCG and OpenAI staff.
Hand a management consultant ChatGPT and a Python problem, and they start doing data science. Take the chatbot away and the new skills vanish. That is the uncomfortable takeaway from a randomized trial of 487 Boston Consulting Group consultants, published today in Nature Human Behaviour.
The paper, titled "Generative AI and the temporary upskilling of knowledge workers", lands in 2026 but describes an experiment from 2024. The consultants used ChatGPT Enterprise, and the post-experiment survey asked them to rate what GPT-4 could and could not do, according to the paper's supplementary materials. The manuscript was received by the journal on 1 August 2024. So this is a peer-reviewed verdict on the GPT-4 generation of ChatGPT, not on the agents and reasoning models companies are selling now.
The researchers gave them data science work outside their skill sets: writing Python to clean and merge messy datasets, spotting errors in a statistical analysis plan, and building a model to score how predictable soccer matches were. The tasks were chosen because ChatGPT could not solve them on its own, so a human still had to steer. One group got access to ChatGPT plus training in how to use it. The control group did not. BCG data scientists sat the same tasks as a benchmark.
The chatbot clearly helped. The authors write that treated workers "scored higher than the control group on three tasks", across coding, statistics and prediction. But the bar the consultants were reaching for stayed out of reach.
"We can reject the hypothesis that they performed as well as expert data scientists." Emma Wiles, Lisa Krayer and co-authors, Nature Human Behaviour
The coding task shows how thin the margin was. The supplementary materials note that only 5 workers in the treatment group got the exact right answer, and none in the control group did, given the time limit. The headline coding score instead measured how many of the required steps workers got right.
The exoskeleton comes off
Then came the part that should worry every executive who has pitched AI as a training program. After the experiment, the researchers quizzed participants on data science and coding questions with no generative AI allowed. Having spent the session working alongside ChatGPT made no measurable difference.
"We find no evidence that treated workers are better at answering technical questions without ChatGPT post-experiment, suggesting that their demonstrated technical capabilities do not imply knowledge acquisition." Emma Wiles, Lisa Krayer and co-authors, Nature Human Behaviour
The authors frame generative AI as an "exoskeleton", one that enhances what workers can attempt, "but these gains depend on continued technology use." Tidy metaphor, awkward implication. An exoskeleton makes you stronger while you wear it. It does not build muscle.
On this evidence, the capability lives in the subscription, not in the staff. For companies weighing how far AI will reshape headcount, a debate that reached Wall Street when Goldman Sachs put a number on displaced workers, the study suggests that cheap AI-assisted "upskilling" may be closer to renting skills than acquiring them.
One more twist. The supplementary tables show that working with the tool raised consultants' confidence that GPT-4 could answer several questions it almost never gets right.
Readers should also know who ran this study. Six of the eight authors, including co-first author Lisa Krayer, are listed at BCG's Henderson Institute. Pamela Mishkin is listed under OpenAI's Economic Impacts Research team, and the paper thanks OpenAI's team for collaboration on task design. The competing-interests statement says the six BCG authors "are or were employees of BCG", that Mishkin "is an employee of OpenAI", and that Boston University's Emma Wiles received consulting fees from BCG while the experiment ran. BCG funded the work through employee time and internal resources.
None of that makes the findings wrong, and it is notable that a paper with OpenAI on the byline concludes its product taught people nothing durable. But it was one firm's consultants, one set of tasks and one generation of ChatGPT. Whether newer models that explain their reasoning do any better as teachers is a question this study does not answer.
