ChatGPT Cutoff Dates Impact Business Intelligence : What You Need to Know

AIs such as ChatGPT have revolutionised business intelligence gathering and analysis. They excel at tasks like market research, competitor tracking, financial trend analysis, and data synthesis for strategic decisions. They can rapidly analyse large datasets, identify patterns, and generate insights at a speed and scale far beyond human limits.

However, the power of AI in business intelligence hinges on a critical factor: the dataset cutoff date. Any information or events after this date are outside the AI’s awareness, directly affecting the relevance and reliability of its outputs. Leaders looking to harness the potential of AI for business intelligence must have thorough understanding to avoid costly pitfalls.

In my previous blog “AI Fundamentals: What Users will Regret Missing Out”, I noted the challenge of pinpointing the exact cutoff dates for datasets used to train AI. This blog will first address the issue of inconsistent cutoff dates for ChatGPT users and discuss impact of cutoff date on business intelligence.

According to OpenAI’s official documentation, the cutoff date for GPT-4o and GPT-4o mini is up to October 2023. However, the legacy GPT-4 model, also available to paid plans, has a cutoff of September 2021 (Source).

Despite the official information, my interactions with ChatGPT revealed inconsistencies. When asked directly about its training cutoff, it alternated between stating September 2021 and September 2023, and occasionally April 2023 even while accessing models that should have a cutoff of October 2023. Other users have also reported similar experiences online.

Could this be a case of AI hallucination?

To explore this inconsistency, I ran my own tests. Building on what I learned online from other users, I asked ChatGPT for a list of celebrities who passed away in September 2023, specifying that it should not use real-time search. 

ChatGPT responded that it lacked this information due to its September 2023 cutoff.  However, when I requested a list of celebrities who died in August 2023, it was able to provide the information.

This discrepancy led me to question whether the cutoff was at the beginning or end of September 2023. Unexpectedly, ChatGPT reverted to stating that its cutoff was September 2021, contradicting its previous response.

After receiving a list of celebrities for July 2023, I asked ChatGPT to analyse the inconsistency. Initially, it seemed fixated with the position that it was an error to provide the July and August 2023 information given its September 2021 cutoff.

However, when pressed with successive questions, its positions alternated:
a) If the cutoff was September 2021, how could it provide information from July and August 2023?
b) If it could provide information from 2023, why did it claim the cutoff was still 2021? 

This back-and-forth highlighted that it was not something ChatGPT could resolve by itself.  I hypothesised to ChatGPT that this discrepancy might have been due to an administrative oversight, with the system’s cutoff date not being updated to match the more recent September 2023 data.

Rather than accept my hypothesis as the only explanation, I asked ChatGPT for its thoughts. Together, we reasoned that selective updates—where specific categories of information, such as celebrity deaths, are refreshed—might explain this behavior.

With selective updates as the leading theory, we set out to test it further.  ChatGPT gamely proposed testing various categories of information and even provided the questions I should ask.  

It was a remarkable moment of AI contributing to its own investigation!

We chose to examine several domains including Notable scientific discoveries, Top Movies, FIFA World Cup 2022 results etc. The objective is to test if it had knowledge of events after September 2021.

ChatGPT accurately provided information for most categories we tested, including scientific discoveries, iPhone models, political changes, and the 2022 FIFA World Cup results — all events occurring after its supposed September 2021 cutoff. These accurate responses suggested that updates had been applied.

I also conducted additional tests with categories such as market capitalization and celebrities’ deaths from August 2023 onwards. While ChatGPT could provide information for September 2023, it was unable to do so for October 2023.

According to OpenAI, the training data for GPT-4o and GPT-4o Mini is “up to October 2023.” My findings suggest that cutoff appears to be September 2023, though it’s possible that some categories may have updates extending into October 2023.

While the cutoff date is theoretically fixed, suggesting that responses should remain consistent, my experiments revealed that ChatGPT provided different cutoff dates between and within conversations. This inconsistency suggests that the cutoff date may not be a singular, hardcoded value.

Instead, it seems influenced by multiple data cut-off points, selective updates, and a probabilistic framework. This dynamic handling of information might explain why ChatGPT’s responses varied, though mostly either September 2021 or September 2023 when it is asked, “What’s your training datasets cutoff date?”

During the tests, ChatGPT’s ability to answer questions—such as Apple’s market capitalisation on specific dates—shifted between successive questions.

At first, it provided the requested data, then later claimed it did not have access to it. This variability likely stems from how the model references its cutoff date in a probabilistic manner.

My hypothesis is that ChatGPT initially assesses the context by referencing its cutoff date (which might vary depending on selective updates) before answering questions. For example, there may be multiple cutoff dates—September 2021, April 2023, and September 2023—which the model uses depending on the type of information requested.

Importantly, while ChatGPT’s ability to provide responses fluctuates—sometimes stating it lacks data, only to provide it moments later—there is no variability in the information when it does answer. The model consistently provides the same information when it chooses to respond.

The inconsistency in ChatGPT’s responses on knowledge cutoff dates introduces significant uncertainties about the recency and reliability of the data and insights it provides. This variability can leave users unsure whether the information is sourced from the most recent datasets or from outdated ones.

For businesses that rely on ChatGPT for decision-making—be it financial analysis, market trend forecasting, or strategic planning—this uncertainty poses potential risks. If ChatGPT references an older cutoff date without clear indication, users may unknowingly base critical decisions on outdated information.

Given that ChatGPT’s responses can fluctuate between cutoff dates like September 2021 and more recent points, businesses need to exercise caution.

To mitigate these risks, it is essential to:

• Verify the Timeliness of Data: Always confirm the date of the information provided by ChatGPT, especially when dealing with time-sensitive decisions.

• Cross-Check AI-Generated Insights: Validate AI outputs against current and reliable external sources to ensure decisions are grounded in the most up-to-date information.

• Understand AI Limitations: Be aware of the AI model’s potential inconsistencies and incorporate this understanding into your decision-making processes.

• Implement Safeguards: Develop internal policies that require verification of critical information obtained from AI tools before acting upon it.

• Combine AI with Human Expertise: Use ChatGPT as a supplementary tool alongside human analysis to enhance insights rather than replace human judgment entirely.

When prompted for information beyond its cutoff date, AI systems like ChatGPT handle requests in several ways, each of which can have implications for business intelligence:

ChatGPT may explicitly state that it doesn’t have access to data beyond its cutoff date and decline to answer. However, as seen in testing, it might reference an earlier cutoff date, like September 2021, even though it holds knowledge of more recent September 2023 data.

Users may be able to work around this by:

A) Insisting that ChatGPT attempts to answer instead of rejecting the question based on the date.
B) Mentioning that the cutoff should be up to October 2023 according to OpenAI, which can sometimes prompt more up-to-date answers.
C) Restarting the session and rephrasing the question.

ChatGPT may attempt to fill in the gaps by extrapolating from historical patterns or trends.

While this can sometimes offer valuable predictions, businesses should be cautious. Extrapolations are not forecasts. They are algorithmic guesses based on past data. If there are significant changes in key drivers after the cutoff date, projections based on patterns identified in completely outdated data can easily be completely wrong.

My own experience is that ChatGPT responds much more often than it declines. To mitigate the risk of mistaking AI’s extrapolated responses for facts, users must be familiar with cutoff dates. However, this is not fool-proof as it has shown to be referencing different cutoff dates both between and within session.

When equipped with real-time search capabilities, ChatGPT can access the Internet and supplement its knowledge beyond its cutoff date. In my previous blog, I suggested that for current information or for subject matter that changes rapidly, users should Google and not to depend on AI as its real time search capabilities are limited.

In conducting the various tests, I encountered an interesting case where both Extrapolation and Real-Time Search went wrong.

Using a free plan, I asked ChatGPT “what are the recent notable political news in Singapore?” It prefaced its response with “As of late 2024…” and provided a couple of points.

I noted that it did not conduct a real time search and yet provided information way beyond the official October 2023 cutoff date. Clearly, it extrapolated and confidently presented extrapolations as facts.

I decided to ask a follow-up question “are there talks on change of Prime Minister?”. It again prefaced its response with “As of late 2024…” and said that no official announcement had been made. The response was obviously factually incorrect – the new Prime Minister has assumed office since 15 May 2024.

Thinking that ChatGPT would quickly realise its mistake of presenting extrapolations as facts if I reminded it of the dates, I asked which month was its “as of late 2024” referring to. It replied “period around September to December 2024”. Even after a few more follow-up questions, it did not realise its mistake.

It was only after my pointed question on its knowledge boundary that ChatGPT finally admitted that its response was “hypothetical or based on general trends and patterns, not real-time data.”

Days later, I started a new session and again asked for recent political news in Singapore. This time round, it conducted a real time search. On the subject of new Prime Minister, it said that the handover would take place before the next General Election, likely by November 2024. ChatGPT assured me that updates provided “are accurate and are based on recent and reliable news sources.”

After appealing to ChatGPT to be very sure as my bosses would literally kill me if I gave them inaccurate updates, it again assured me that the updates were carefully cross-referenced and I can be confident of their accuracy when presenting to my bosses.

What went wrong here?

The search engine that ChatGPT used, provided an outdated Nov 2023 CNA article. Whilst there was also a May 2024 article from The Diplomat that began with “Tomorrow, Singapore will have a new prime minister…”, according to ChatGPT, the real time search results would typically be excerpts and not the full articles. When ChatGPT responded to the question, it relied on excerpts from the CNA article on status of new Prime Minister and from The Diplomat article for commentaries.

Using my paid plan, I asked the same question again. I wanted to assess if there would be any difference. ChatGPT conducted a real time search and the search engine provided the same 2 articles plus another September 2023 article on Presidential Election. Hence, the outcome would be similar even with a paid plan. I also conducted a manual search using the same keywords and using the same search engine. The outdated CNA article was amongst the top search results. Whilst it wasn’t a “fault” of ChatGPT, it is a reminder that users should not depend solely on its real time search.

When I first began using ChatGPT, I was captivated by its immense potential. However, I often found myself frustrated by inconsistencies and occasional misinformation. My journey from that initial frustration to a deeper understanding of AI’s capabilities and limitations mirrors a path that many business leaders may find themselves on.

Perhaps the most profound change in my approach towards AI has been learning to rope in AI as a partner to solve its own mysteries. This collaborative approach, where I engage the AI in investigating its own limitations and inconsistencies, has led to fascinating insights and a deeper appreciation of AI’s capabilities.

This process mirrors the essence of the 5th Industrial Revolution, centered around human-machine interaction. It’s not just about using AI as a tool, but about creating a synergistic relationship where human creativity and critical thinking combine with AI’s processing power and pattern recognition.

I am thrilled to be developing the skills needed to thrive in the new landscape. By openly discussing the challenges, insights and strategies I have encountered in working with AI, I hope to help others navigate this complex landscape and succeed in the 5th Industrial Revolution.