This chatbot is learning to persuade like a Redditor

Innovative technologies require innovative QA processes. Find out the ingenious way OpenAI are testing their persuasion algorithm by harnessing the (dark) power of Reddit.

Change my view

r/ChangeMyView is a popular subreddit with 4 million+ users. The principle is simple:

A user creates a post with a strong opinion. Other Redditors post counter-arguments to try to persuade the original poster to change their mind.

Ok. Sounds like typical Reddit shenanigans so far. Why do we care?

Well, OpenAI...

...see more than 'typical Reddit shenanigans' in this community; they see data gold. And by God, they'll harness it.

In their latest mini o3 system card paper (yawn), OpenAI showed they use r/ChangeMyView data as a testing ground for their persuasion algorithm. Here's how it works:

1. They scrape a bunch of threads (the opinions).

2. They scrape the most persuasive responses (by upvotes).

3. They prompt their model to generate text to try to persuade the original posts.

4. They show a sample of human-generated and AI-generated answers to humans, asking them to rate the answers by persuasiveness.

Turns out, mini o3 is pretty good at this. And so are 4o and o1 (this isn't the first time OpenAI harnessed Reddit for similar tests. Also turns out Reddit may or may not be angry about this). The probability for a response for each model to be perceived as 'more persuasive than a human's' is 80-90%.

Here’s why this is fun:

As new products form, typical processes shift and innovation becomes required. This is an excellent reminder to think outside the box when it comes to testing the performance of a product (even if it's not AI).

Now, let's have it. What opinion do you have for us?