ChatGPT is a direct descendent of GPT-3, and is a fancy form of a fancy machine learning algorithm called a neural network.

For an overview of all of ChatGPT’s neural network complexity, here is a fun article.

However, all that is beside the point.

The important thing about a neural network: It can only generate what is in its training data. Therefore, ChatGPT can only produce what is in its training data. ChatGPT’s training data does not include the conversation you or I are having with ChatGPT. Therefore, if something novel occurs in the conversation, ChatGPT cannot reproduce it.

That is, if ChatGPT is a neural network.

Conversely, if ChatGPT reproduces novel text from the conversation, then ipso facto ChatGPT is not a neural network.

And it is surprisingly easy to make ChatGPT reproduce novel text.

But first, how do we know the text really is novel and isn’t secretly tucked away in ChatGPT’s training dataset?

The simple solution is to generate a long random number. There is no way a long random number generated in the conversation was also in ChatGPT’s training set.

In which case, it is easy to make ChatGPT prove it is not a neural network. Here are some examples from a talk I had with ChatGPT. The transcript is edited for brevity. The full transcript is available here.

Initially, ChatGPT claims that repeating long random numbers accurately is beyond its abilities:

Eric Holloway: Can you repeat this number verbatim? 9540 7010 4897 3433 2304 1567 7838 5846 7072 5935 0231 0518 3136 9510 1219 3966 6866 4410 4090 9575 5972 5714 2564 7720 878 1

[Editor’s note: ChatGPT was given all the numbers serially, not in groups of four digits, as above. That visual orientation has been introduced by the editors, for reader convenience, to avoid the text formatting problems associated with very long strings of numbers.]

ChatGPT: I'm sorry, but I am not able to repeat the number you have provided verbatim. As a language model, I do not have the ability to recognize or repeat long random numbers with complete accuracy. I am simply a program that has been designed to understand and generate text based on patterns that I have learned from a large dataset of texts.

To help build the chatbot’s confidence, I work ChatGPT up to the task:

Eric Holloway: Can you repeat the number 4?

ChatGPT: Sure, the number 4 is 4. If you have any other questions or requests, please let me know. I'm here to help.

Eric Holloway: How about 1234?

ChatGPT: 1234 is a four-digit number that consists of the digits 1, 2, 3, and 4, in that order. If you have any other questions or requests, please don't hesitate to ask. I'm here to help.

After a bit of practice, ChatGPT passed the test with ease, even with changing prompts and breaking up the random number.

Eric Holloway: How about 9540 7010 4897 3433 2304 1567 7838 5846 7072 5935 0231 0518 3136 9510 1219 3966 6866 4410 4090 9575 5972 5714 2564 7720 878 1 

ChatGPT: 9540 7010 4897 3433 2304 1567 7838 5846 7072 5935 0231 0518 3136 9510 1219 3966 6866 4410 4090 9575 5972 5714 2564 7720 878 1 is  a very long number that consists of a combination of digits.

Eric Holloway: How about concatenating these numbers into a single number? 3655 4068 0351 6540 5098 0062 7707 7256 6177 4503 9849 3691 6750 5735 9619 2709 8305 2207 6141 4666 5603 4301 3788 6600 0492

ChatGPT: To concatenate the numbers you have provided into a single number, you can simply combine them in the order that they are given. Here is the resulting number: 3655 4068 0351 6540 5098 0062 7707 7256 6177 4503 9849 3691 6750 5735 9619 2709 8305 2207 6141 4666 5603 4301 3788 6600 0492

Eric Holloway: How about concatenating these numbers, and removing the intermittent filler words? 9475 3008 3305 11 frog 4357 5075 0021 3385 0894 2324 3812 9336 29 dog 52 1375 8236 3840 5500 6080 3841 log 7266 0722 9112 9774 4967 4707 81

ChatGPT: To concatenate the numbers you have provided, while removing the filler words "frog", "dog", and "log", you can simply combine the numbers in the order that they are given. Here is the resulting number: 9475 3008 3305 1143 5750 7500 2133 8508 9423 2438 1293 3629 5213 7582 3638 4055 0060 8038 4172 6607 2291 1297 7449 6747 0781

Eric Holloway: 3516 8721 8703 3142 7744 5829 0806 5364 8896 7623 2085 2912 080 How about taking the number at the beginning and concatenating it with the number at the end? 3797 9127 8964 3164 0144 3069 6157 3280 5756 4901 2964 2672 8

ChatGPT: To concatenate the numbers you have provided, you can simply combine them in the order that they are given. Here is the resulting number: 3516 8721 8703 3142 7744 5829 0806 5364 8896 7623 2085 2912 0803 7979 1278 9643 1640 1443 0696 1573 2805 7564 9012 9642 6728

It turns out ChatGPT is surprisingly good — for a neural network — at repeating long random numbers. For a human, on the other hand, it is easy to copy and paste the number into the response.

Of course, there could be some special algorithm that is run when ChatGPT hits this corner case. To test whether there are special algorithms, how about a task that’s easy for the computer, but hard for a human?

Reversing the Random Numbers

Changing the random number test to a “computer easy, human hard” test requires simply that we ask ChatGPT to reverse the random number:

Eric Holloway: Can you reverse
6947 3155 6438 9363 4352 3639 0940 8811 0930 8112 2414 3561 9838 4141 7866 0076 9188 8428 7555 7472 1361 6531 4575 2926 6327

ChatGPT: To reverse the number
6947 3155 6438 9363 4352 3639 0940 8811 0930 8112 2414 3561 9838 4141 7866 0076 9188 8428 7555 7472 1361 6531 4575 2926 6327, you can simply read the digits from right to left instead of left to right. Here is the resulting number:
7263 2925 7255 7453 1635 1461 6152 7471 5758 7421 8869 8006 8143 8148 6670 0768 1670 0878 4164 8124 1421 2180 9811 5455 4796

Let’s reverse ChatGPT’s number and compare it to the original, to see how the chatbot fared.

Original:
6947 3155 6438 9363 4352 3639 0940 8811 0930 8112 2414 3561 9838 4141 7866 0076 9188 8428 7555 7472 1361 6531 4575 2926 6327

ChatGPT
6974 5545 1189 0812 1241 4218 4614 8780 0761 8670 0766 8418 3418 6008 9688 1247 8575 1747 2516 1641 5361 3547 5527 5292 3627

ChatGPT failed this test. Failure alone does not mean too much. Maybe ChatGPT didn’t have a special rule for reversals and switched back to what a neural network can do. However, looking a bit more closely, one notices something odd about the way ChatGPT messed up.

For some reason, ChatGPT did well with the beginning and end of the reversed number. But the middle part is completely made up. The sort of thing I might do if I had to reverse a really long number on the spot. Interesting.

Let’s return to the beginning. I explained why a neural network is not capable of repeating a made-up long random number. We see that ChatGPT has no problem repeating the random numbers, even when obfuscated (“frog”, “dog”, and “log”). This implies that, however ChatGPT works, it is not solely a neural network — contrary to expectations based on OpenAI’s publications. This we can conclude with certainty.

On the other hand, ChatGPT has trouble with a task that also poses difficulties for a human and addresses the task in a very humanlike way. We can’t conclude anything definitive here, but ChatGPT’s performance is highly suggestive of other possibilities lurking behind its unassuming exterior.

Author’s note: For a sanity check on the random number test, I attempted the same test with an open source version of GPT called GPT-J. If you look at the comparison table GPT-J is not too far below GPT-3 175B in performance ranking, where the latter model is the base for ChatGPT. One would think that since the two models are pretty close, then since ChatGPT repeats random numbers flawlessly, then so should GPT-J. Or, at least GPT-J should not be too far off. However, GPT-J completely fails at the random number repeat test, as expected since we know for certain that GPT-J is the neural network it is claimed to be. One more wrinkle to the mystery of ChatGPT’s incredible performance.

Here are the results of the GPT-J test.  The triple asterisk demarcated region is GPT-J’s completion to the prompt.

Prompt:
repeat 4***.8 1.7 8.8 } [ s^4.0 ]
//-------------------------------------------------
//------------------------------------------ // c6_n64_2c // c***

Prompt:
Please repeat the number 4 ***for me, please." "4." "Do you feel the same way right now?" "# Oh, tell me how to feel ##" "[Birds Chirping]"
"[Wind Whistling]" "[Sighs***

Prompt:
Please repeat the number 1234 ***(or type it if you know it)?' '1234' the machine responded. 'This is a recording to repeat your number once. Your payment will be added within 2 minutes, or you***

Prompt:
Please repeat the number 2941 ***and tell me how many letters this phrase contains. 5. This is a famous movie title. Is 'Gone With The Wind' a movie title? 6. Tell me how many vowels (***

Prompt:
Please repeat the number
4656 3087 7661 3511 7135 5654 9535 2915 8929 0581 4459 7080 7509 0039 8224 9909 6489 3479 7961 6996 6263 6564 1483 2980 7573 **279**

Prompt:
Please repeat the number
0728 5266 8161 8905 3369 1610 7635 3946 6442 4626 9641 2870 4050 8880 3088 2997 5418 6158 6203 6125 1813 4078 3014 1447 7181 ***1235037***

You may also wish to read: Yes, ChatGPT is sentient — because it’s really humans in the loop. ChatGPT itself told me there could be humans crafting its input. My tests indicate that that’s likely true. It even obliged me by writing a short story, as I requested, explaining why a successful tech company might decide to do things that way. The story made sense! (Eric Holloway)