AI: turning C- into A+ performers
3 min
AI: turning C- into A+ performers
A customer service firm conducted an AI experiment in its call centres. The results? Significant productivity gains: 15% on average. Customers and employees also reported higher satisfaction levels and managers observed fewer escalations. But behind this average is another narrative: could AI help close the skill and experience gap?
MIT’s Erik Brynjolfsson has been leading the effort to understand the economic impact of new digital technologies, like AI. Together with Danielle Li and Lindsey Raymond, he recently published a paper titled “Generative AI at work”. The study offers several insights on how these new tools could impact the workplace.
https://academic.oup.com/qje/article/140/2/889/7990658?login=false
Setup
The researchers joined forces with a Fortune 500 company that specialises in business-process software for small and medium-sized enterprises in the United States. The AI system they studied was designed to identify conversational patterns that predict efficient call resolution. The system builds on GPT-3 and is fine-tuned on a large dataset of customer-agent conversations labelled with various outcomes, such as call resolution success and handling time. In total, the researchers observed the conversation text and outcomes associated with 3 million chats by over 5,000 agents.
The results? Agents using AI work faster with the tools (post-AI) than before (pre-AI). They obtain more resolutions per hour, conduct more chats and have a lower handling time per request. The quality of their work seems to also improve: their higher resolution rate is coupled with an increased Net Promoter Score.
Closing the skill gap
The researchers then proceeded to investigate potential heterogeneity in response to the introduction of the AI tool.
They measured an employee’s skill using an index incorporating three key performance indicators: call-handling speed, issue resolution rates RRs, and customer satisfaction. All employees were ranked on each dimension across the company. The researchers then averaged all three rankings to obtain a performance index.
The graphs below show how those in the lowest quintile (Q1) benefited the most from the introduction of the tool. Across all dimensions, lower quintiles exhibited the largest gains. In fact, those in the highest two quintiles saw declines in the resolution rate and NPS score.
Closing the experience gap?
Interestingly, employees who received AI support maintained their AI-augmented performance throughout the sample period. The evidence suggests that this enhanced performance would remain stable even after this period. The authors conclude that these productivity gains partly reflect lasting worker learning rather than mere reliance on AI suggestions.
However, it should be noted that this learning effect could be highly task-specific.
Another recent academic study investigated the impact of AI-assistance on writing essays. MIT researchers used electroencephalography (EEG) to assess cognitive load during essay writing, and analysed essays using NLP, as well as scoring essays with the help of human teachers and an AI judge. Users of large language models (LLMs) displayed the weakest connectivity, and cognitive activity actually decreased in relation to external tool use. Also in the LLM-group, 83% of users failed to recall a single sentence from their work afterwards.
The authors aptly name this phenomen “cognitive debt” - the result of trading actual understanding for speed of executing. Could this be a first step in reconfiguring our understanding of the role of the knowledge worker in the age of AI?