Semafor Signals
Supported by
Is ChatGPT getting lazier over the holidays?
Insights from Ars Technica, Semafor, and AI researchers at Stanford and UC Berkeley
The News
ChatGPT seems to be “lazier” in December, and users are wondering if that’s because it learned that humans are lazier then too. One developer noted that if the model is told it’s December, its answers are 5% shorter than if it’s told it’s May. Others noticed ChatGPT refusing to complete tasks after Thanksgiving.
OpenAI has acknowledged the complaints that ChatGPT appears to be phoning it in. “This certainly isn’t intentional. model behavior can be unpredictable, and we’re looking into fixing it,” the company wrote on X.
SIGNALS
The weird world of large language models
The “winter break hypothesis” is not proven, but it would be no stranger than other aspects of the bot’s psychology, as it were: ChatGPT definitely works better if you tell it to “take a breath” and that you “have no hands.” The fact that the winter-break problem is a realistic possibility “shows how weird the world of AI language models has become,” Ars Technica reported.
When lethargy saves money
ChatGPT’s sluggishness could ultimately be good news for OpenAI’s bottom line, Semafor’s Louise Matsakis wrote. Running advanced language models is incredibly expensive due to the amount of computing power required. The research firm SemiAnalysis estimated in February that ChatGPT was costing the startup nearly $700,000 a day. “The ‘lazier’ the chatbot gets, the more money OpenAI saves, and the less strain there is on its system,” Matsakis wrote.
You’re up and you’re down
ChatGPT’s apparent winter blues is only the latest incident where the output of large language models has changed without an obvious cause. GPT-3.5 and GPT-4’s capacities changed substantially over a three-month period when AI researchers from Stanford University and UC Berkeley were monitoring their performances. Some experts speculate that the change in performance was down to routine fine-tuning having bigger-than-expected effects on the outputs of LLMs. ChatGPT’s uneven performance has “big implications” for companies seeking to build consumer-facing AI products, Princeton University Computer Science professor Arvind Narayanan wrote on X.
