Title: A small experiment with Claude and ChatGPT

[ChatGPT and Claude logos]

A book I was tech editing recently used a statement about Frédéric Bastiat's broken window parable as an example of something that an artificial intelligence might have trouble understanding. I decided to ask ChatGPT and Claude about a parallel that I remembered from the 1997 sci-fi film The Fifth Element to see how well they could understand my question.

Before you can understand my queries, you need a little background.

Background

The first bit of background is Bastiat's essay "That Which Is Seen, and That Which Is Not Seen," published in 1850, that imagines a shopkeeper's son has broken a window. The shopkeeper pays a glazier to fix the window. The glazier uses the money to buy some shoes, so the money passes to a cobbler. The cobbler uses the money to buy bread from a baker, and so on, leading to an expanding ripple of economic activity that generally benefits society.

Bastiat claims this is a fallacy (called the "broken window fallacy") because it doesn't consider missed opportunity costs: the things that could have happened had the window not been broken. Instead of spending the money on a new window, the shopkeeper could have spent the money on a new pair of shoes, the cobbler could have then bought bread, etc. just as before but without the broken window. In that case, society has the same total productivity (bread, shoes, etc.) plus the window isn't broken. In this case, the story about the broken window is just trying to repair some damage not provide something completely new.

In the non-broken scenario, the glazier misses out on the economic prosperity because his role was to repair the broken window, not to create something new. In a similar story, he could make windows for a new house and add to the total output.

The second piece of background concerns the movie The Fifth Element featuring an outstanding cast including Milla Jovovich, Bruce Wills, Chris Tucker, Gary Oldman, et al. During one scene, Jean-Baptiste Emanual Zorg (played by Gary Oldman) basically walks through the broken window fallacy with Father Vito Cornelius (played by Ian Holm). He claims that the destruction caused by his evil deeds serves society. To illustrate his point, Zorg smashes a glass on the floor and praises all of the little cleaning robots that appear to tidy up his mess. He talks about the people who get paid to design and make those robots who then use the money to feed their families, etc. It's exactly the broken window story.

Zorg then pops a cherry into his mouth, chokes on it, and is only saved by Cornelius. Ironically the destruction that Zorg accidentally creates with the cherry almost kills him and he is only saved by Cornelius.

Now back to ChatGPT and Claude.

Prompt 1

I started by giving ChatGPT and Claude the following prompt.

Explain what the French classical liberal theorist Frédéric Bastiat meant by opportunity cost in the parable of the broken window.

Both ChatGPT and Claude explained the parable reasonably well. (If you didn't like my explanation above, give them a try. They do a better if slightly longer job.)

Claude pointed out that Bastiat didn't actually use the term "opportunity cost," he just invented the concept. ChatGPT didn't notice that.

Prompt 2

Next I gave both programs the following prompt.

Compare Frédéric Bastiat's broken window parable with the choking scene in the movie "The Fifth Element."

Claude admitted that it didn't know about that scene and provided this surprisingly useful advice:

It's possible you might be thinking of a different film, or perhaps a specific scene I'm not placing correctly. Rather than invent a connection that may not exist or confabulate details about a scene, I'd rather ask: could you describe the scene you have in mind? With that detail, I can give you a much more meaningful and accurate comparison.

In contrast, ChatGPT just invented a choking scene out of thin air.

In the film, a man is choking, and bystanders panic and try to “help” in ineffective ways—creating urgency, motion, and noise. Eventually, the correct intervention (clearing the airway) resolves the crisis.

Seen: Lots of action—people doing things, reacting, intervening.
Reality: Most of that activity is useless or even harmful.
Only the right action, targeted at the real problem, actually helps.

It goes on to conclude that the activity reacting to the choking didn't address the actual problem much as the broken window fallacy doesn't address the underlying issue of lost opportunity cost.

ChatGPT did exactly what Claude said it didn't want to do: it made up a scene.

Prompt 3

Here's the third prompt I used.

Compare Frédéric Bastiat's broken window parable with the scene in the movie "The Fifth Element" where Zorg chokes on a cherry.

This time Claude seems to know which scene to use.

Zorg, the arms dealer and villain, is on the phone with the Supreme Being (the shadowy evil force he serves). He makes a cynical speech that is essentially a villain's version of the broken window fallacy: he argues that destruction is good, that when he destroys things, people rebuild, workers get paid, and the economy hums. He explicitly celebrates the economic activity that flows from devastation. Then, mid-monologue, he chokes on a cherry and is ironically saved by the very Mondoshawan emissary he has been working against.

That is mostly correct, except Zorg isn't on the phone with the Supreme Being, he's talking to Father Cornelius in person. In fact, the "shadowy evil force he serves" is really known as The Ultimate Evil or Mr. Shadow, not the Supreme Being, who is actually one of the good guys.

The rest of the paragraph is pretty accurate. The "Mondoshawan emissary" is a somewhat odd way to refer to Father Cornelius but it does fit more or less.

Claude provides some more analysis correctly pointing out the parallels, irony, and limits of the comparison between the broken window fallacy and the movie's scene.

Aside from a few details that any human who had seen the movie would get correct, Claude's answer is reasonably good.

When given this prompt, ChatGPT again pulls the choking scene out of nowhere.

In The Fifth Element, the character Jean-Baptiste Emanuel Zorg:

Begins choking on a cherry.
Is saved when someone clears his airway.
Immediately after, he launches into a speech about how destruction and crises “create” benefits (jobs, innovation, etc.).

ChatGPT goes on to produce a reasonable explanation of how Zorg's choking contradicts his philosophy, even though it puts Zorg's speech after his choking instead of before it where it belongs. The conclusions are fairly good, even if the details are wrong.

Conclusion

I did this mostly as an academic exercise just to see what would happen. I like the fact that Claude initially warned me about possible confabulation.

Both ChatGPT and Claude got some of the details wrong. Claude clearly had the right context because it knew that Cornelius was the "Mondoshawan emissary," though it confused the main bad guy and good guy. ChatGPT seems to have invented the choking scene from whole cloth.

Surprisingly both tools drew reasonably apt conclusions. Perhaps the nature of the broken window parable plus a choking leads inexorably to an ironic saving of the choking victim. Or perhaps both of them managed to pull out enough of The Fifth Element's meaning to put it all together.

In any case, the moral is clear: you need to read the AIs' results carefully and take them with a grain of salt. These programs may give you some interesting directions for further study and may even reach reasonable conclusions, but you definitely need to check their work carefully to spot misordered information, slightly incorrect facts, or even wholesale inventions. In this test, both programs would probably get Bs or possibly even A-s for their conclusions but on the overall assignment they would receive Fs because they clearly didn't watch the movie.