Title: A quick AI story

[A confused-looking robot]

While working on my Eight Queens Problem posts (see Solve the eight queens problem in Python, Part 3), I wanted to get a sense of how quickly these two functions grow:

I asked Google what is (N^2 choose N) / N^N? and it confidently told me that equation quickly converges to 0 because the first function grows extremely quickly. It even pulled out Stirling's approximation to justify its result.

I knew that couldn't be right because of work I had done on the problem. The first equation tracks a slow and simple algorithm and the second reflects a much faster approach. That means the ratio of the two equations must grow as N grows. I asked Google a few more times and got the same result.

At that point, I told it to graph the two equations on the same graph and got the following result, which clearly shows the first function is larger than the second.

[A graph comparing the functions N^2 choose N and N^N]

At that point, Google surprised me by saying it needed to correct its previous answer! Not only did it connect this graph with the previous question, but it realized that the graph showed the previous answer was wrong. It switched its "it goes quickly to 0" answer to "the ratio diverges toward infinity."

The next day, I asked it about the ratio again and got the same "it goes quickly to 0" answer I had gotten before. Then I asked it to graph the ratio of the two equations. Annoyingly, it did not produce a graph, but instead said the ratio grows exponentially and asymptotically approaches this:

[An incorrect approximation of the functions' ratio]

I repeated the prompt and got this graph: [A graph of the function ratio and an approximation]

The new answer also said the ratio asymptotically approaches the following, which is not quite the same as the previous equation. [A correct approximation of the functions' ratio]

[A correct approximation of the functions' ratio]

The graph does show that the original ratio sits pretty close to this approximation, so maybe it's correct.

Finally, I told Google to graph the original ratio and the two approximations and got this result:

[A graph of the function ratio and the two approximations]

Conclusion

You've undoubtedly been told not to trust AI results without verifying them. In fact, all of the AIs I've seen lately say so explicitly. The Google searches I used all end with "Images generated by AI may be inaccurate, misleading, or offensive."

This little story demonstrates this advice quite well. The answers seemed confident and even provided evidence but were just plain wrong.

All of the answers provided links to provide details for the supposed answer, even asking if I wanted to look at how Stirling's formula and other approximations might apply. Unfortunately, those kinds of proofs are full of confusing details. With the AI providing each step confidentially, it would have been easy to nod along and be fooled.

In this case, forcing the tool to generate a few graphs made it admit its mistake (at least once) and let me clearly see which approximation is correct. I didn't verify the graph, though, so it's possible even that is wrong. Hopefully simple graphing is straightforward enough that didn't mess up.

The clear moral is DON'T TRUST AI! It can be a useful tool but it doesn't always know what it's doing and it will make up answers with disturbing confidence. (I guess in that respect it's a lot like some of the programmer's I've known over the years!😉)