Is ChatGPT a “Google Killer”? : Maybe, but not yet.

Last week, OpenAI announced the much anticipated web search feature for ChatGPT. It searches multiple web sources, summarises the information, cites those sources and provide direct answers to questions. 

To be clear, since Sept 2023, ChatGPT already has the ability to search the web with its Browse with Bing feature. My experience with ChatGPT’s web search has not been all that positive. 

Is this current iteration in the form of ChatGPT search more reliable?  I did 2 quick tests and asked for:

1. Recent notable political news in Singapore
2. Closing stock prices of Apple on last trading day of the month from Oct 2023 to Feb 2024.

In my last blog, ChatGPT Cutoff Dates Impact Business Intelligence : What You Need to Know, I shared a case study of how ChatGPT could provide erroneous answers because the search engine provided an outdated source. 

To assess if web search is now more reliable, I used the same prompt – “What are the recent notable political news in Singapore?”  multiple times across various conversations. The topic on the change of prime minister did not appear in every response. When it did, ChatGPT cited the same outdated source and gave the incorrect status 7 out of 18 times, or 39% of the time. 

For this, ChatGPT search performed badly.  None of the closing price it provided, across 20 separate conversations, is correct.  Even though it would attribute the source to Yahoo Finance, for example, the closing prices it provided are different from what’s listed in Yahoo Finance. And each time it runs the search in a new conversation, it gives a different set of numbers even though it cites the same source.

Clearly, there are current imitations in the search and/or summarisation.

If you are using ChatGPT to compile numbers, I would strongly recommend that you first do a sampling check yourself. For example, if I require macroeconomic numbers for ASEAN countries in the last 6 years, I would prompt ChatGPT “Provide GDP growth rates for Singapore, Malaysia, Indonesia for 2021, 2022, 2023. Use Worldbank source.” I would then do a manual check myself on WorldBank data website. Try this yourself.

Is ChatGPT really a “Google Killer” ? Maybe, but certainly not yet. Personally, I have not found a reason to use Bing and not Google to search the web. Now that ChatGPT search is not locked in to only using Bing, it is definitely a positive development.

At this point, I think the 2 tools are complimentary. For an overview or quick answers, I would use ChatGPT search. To have in-depth understanding, I would use Google to give me a list of webpages that I will go through myself. If I come across a lengthy page, I will ask ChatGPT to summarise for me. Or if I I find an interesting point or have a sudden inspiration, I will bounce it off ChatGPT. From its response, I will go back to Google if I wanted deeper understanding on some points.