Good Luck Beating ChatGPT at Hangman

Good Luck Beating ChatGPT at Hangman

If you want to know why it's probably not a good idea to use generative AI in financial services (yet), play hangman with ChatGPT.

Here's what happens.

It starts out normally enough. You ask it to play. It explains the rules and then generates a word.

My word had 9 letters and ends in an “o”.

No alt text provided for this image

Fun challenge!

But then it gets weird.

ChatGPT forgets that there are two “a”s in the word until I remind it what it had just told me.

No alt text provided for this image

Then it tells me there is an “s” in the word, but then it changes its mind and says there is no “s”.

No alt text provided for this image

So then I have to remind it (multiple times!) that because I guessed wrongly, I get a body part.

No alt text provided for this image

It apologizes and eventually punishes me appropriately.

I then make multiple additional wrong guesses (what the hell is this word?) and get lied to again, this time about the number of “e”s in the word.

No alt text provided for this image

Finally, I run out of guesses, and it wins.

The word was “Weatheral”.

No alt text provided for this image

I didn’t think you were allowed to use names in hangman, but apparently, ChatGPT doesn’t play by those rules. It reassures me that even though it’s uncommon, “Weatheral” is a real surname.

No alt text provided for this image

HOWEVER!

You will notice that ChatGPT cheated at several points along the way, telling me the wrong places for certain letters and just removing the “o” from the end altogether.

No alt text provided for this image

This isn’t good enough for me, so I ask it to diagnose what went wrong — why did it give me this nonsense word that didn’t match the letters I’d already guessed?

The answer is fascinating.

No alt text provided for this image

It generated the word *after* I’d already made all my guesses!

That’s the opposite of how hangman works!

But large language models (LLMs) are predictive. They’re designed to tell you what they think you want to hear, not what’s objectively true.

It worked as designed.

And that’s the problem with using generative AI techniques like LLMs and transformers in financial services.

You don’t want a chatbot guessing what someone’s credit limit is or telling a customer what it thinks they want to hear when collecting on a debt.

We need certainty in financial services.

If you want to learn more about how generative AI works and (maybe) where it actually might be useful in financial services, read my latest essay over at Fintech Takes: The Most Intriguing (and Terrifying) Fintech Use Case for Generative AI.


Bradley Safer

Technology Leader / Strategy & Architecture / Banking & Insurance

1y

AI is based on probabilistic outcomes, hangman is most definitely deterministic. Had you asked ChatGPT to guess the word you were thinking of, I suspect it would have resulted in quite a different and more accurate gaming session.

Jeff Schultz

Strategic FinTech Partnerships I Payments I Card Guru & Advisor

1y

This was great!

Like
Reply
Steven Kravit

Startups | Fintech | Banking as a Service | Payments | Technology Strategy | Product Leadership

1y

Thanks for sharing, that was very informative.

Like
Reply
Khaleel Rahmaan

Empowering Enterprises with SaaS Solutions

1y

I wonder how it would text the email for an AA(Adverse Action) Notice to the customer. Curious to try.

To view or add a comment, sign in

Insights from the community

Others also viewed

Explore topics