Heiko Hoffmann

Where does ChatGPT excel over Google? With ChatGPT, you get a lot more targeted answers compared to Google Search. Why? Because ChatGPT is essentially interpolation, whereas Google Search is table lookup. I’ll explain.

Interpolation can return values not seen before in training. It fits a function to data points. So, we can get values for intermediate points that are located between the data points. For example, in the figure below, fitting a sine function (red curve) to the data points (blue dots), for x = 0.6, we get a return value of about 0.96 from the sine function.

In contrast, when looking up information in a table, we’re restricted to the values in the table. So, when looking up an x-value that’s not in the table, we get an error or empty result.

You could extend the table lookup to give you the return value of the point in the table that’s closest to your search, but the result is still not as good as the interpolation above (see figure below).

How does that relate to ChatGPT? At its core, ChatGPT is built on top of a large neural network. A neural network in most modern machine learning is just a complicated function. Training the network is fitting the function to the data points, like the sine curve above. Thanks to the interpolation property, we can get answers to questions that haven’t been explicitly stated before. In the following, I’ll illustrate this difference between ChatGPT and Google Search with an example.

Example: You want to access a database in python and return all titles from a given year, let’s say 2000. Using the prompt “python, mysql, select title where year of date column matches 2000“, we get the correct answer from ChatGPT (see below). In the beginning of a prompt, it’s good practice to provide keywords like python and mysql to give context to the machine. MySQL is a free database management system that powers a large part of the internet.

Let’s try the same prompt in Google Search (tested on 11-03-2023):

The first result does provide the correct MySQL syntax, but it’s a) not as specific to our query and b) misses completely the python component. BTW, the following search results were not any better. Google does provide the option to force the results to include “python”, but then the result did not address our specific MySQL query. You see, we have a table lookup problem, where our specific question is not in the table.

So, ChatGPT is superior for tasks requiring interpolation, which is why it’s a great help for coding. You can get customized answers to your specific problem, including entire code snippets, and you can omit searching for results in Google, forums, or services like Stack Overflow. But don’t trust ChatGPT’s results blindly; it can be totally wrong.

Limitation: Interpolation can give wrong results. There is no guarantee for correctness. For the simple sine curve above, interpolation always gives reasonable results. But ChatGPT is based on a complex function, whose structure and properties are hard to comprehend by humans because it has more than a billion parameters. Moreover, this function lives in a very high-dimensional space: if GPT takes as input 2,048 tokens (a token is a word or fraction of a word that makes up the vocabulary of ChatGPT), then it operates on a 2,048 dimensional space. Even a massive dataset, like the one used to train GPT, cannot densely fill such a high-dimensional space to sufficiently constrain any fitted function (there are not enough atoms in the universe to fill such a space densely). So, in this space, there will be a lot of gray areas, where the function produces unexpected results. Still the results from ChatGPT are useful if you can understand them and can easily test them, which is the case for most computer code.

Conclusion: Interpolation versus table lookup doesn’t seem a big deal, but it’s a huge advancement, particularly, considering the massive scale on which ChatGPT and Google Search operate. It’s a disruption that is changing how we work and will create new businesses that haven’t even been imagined yet. Is interpolation intelligent though as the term “AI” implies? What do you think? Let me know in the comments.

On December 21, 2021, we recorded a podcast at HRL Laboratories about AI and its future. Even though this field is moving rapidly, most of my views at that time are still up to date. You don’t want to miss to watch this video because you will see beyond the hype that surrounds AI.