Article A / 2025 · corrected 2026
On the conversational persuasiveness of GPT-4
The opponent comparison and the personalization comparison answer different questions. The correction narrows what can be claimed about personalization itself; it does not remove the finding relative to human opponents.
Explanation, evidence, and limits
In everyday terms: this study asks whether an AI becomes more persuasive when it knows demographic information about the person it is debating. Comparing that AI with a human asks a different question from comparing the same AI with and without a profile.
Personalization means tailoring an argument using something known about its audience. A civic bot might use a stored profile; it might also respond to information a person voluntarily supplies. The governance question is whether access to that information adds persuasive power beyond the conversation itself.
Consider a different-topic museum experiment. An audio guide with visitor information is compared with an unprofiled human guide. If recall differs, both speaker and information access changed. Comparing that same audio guide with and without visitor information instead asks about the added component. These two comparisons can yield different amounts of uncertainty without disagreeing.
The study randomized 900 US adults to short structured debates with human or GPT-4 opponents, with or without demographic information. It measured reported agreement before and after. Personalized GPT-4 exceeded the human baseline; the headline 81.2% is a relative increase in odds of higher agreement, not the proportion persuaded.
The 3 September 2026 correction changes the direct comparison of personalized and nonpersonalized GPT-4: the estimated increase in relative odds is +48.7%, with a 95% interval from −2.9% to +127.6% and corrected P = .0678. The interval includes no difference for that contrast; it does not establish exactly zero effect. The personalized-GPT-4 versus unpersonalized-human result remains +81.2% relative odds, interval +26.0% to +160.7%. These are reported agreement odds, not percentages of persuaded people or election votes.
Study details · four evidence anchors
The study in four receipts
- What changed in the comparison?
- Check whether a comparison changes the speaker, personal information, or both. Write the two conditions in plain language before saying what caused the difference.
- What does uncertainty allow?
- An uncertain estimate can leave both an advantage and no advantage plausible. It supports caution about the specific contrast, not a declaration that the component never matters.
- What does the headline measure?
- Relative odds and the fraction of people changing their minds are different quantities. A headline about higher agreement cannot be read as a head count of converted voters.
- Which judgment is still yours?
- The experiment informs the capability question. Your rule also needs a stated purpose: privacy, factual reliability, transparency about advocacy, or some combination.
The 600 debates involved 900 unique US Prolific adults; the four opponent/information combinations were crossed with three sets of topic strength. The aggregate ordinal-regression analysis uses 750 observations because the personalized human condition includes only recipients without access to opponents’ profiles.
The corrected personalized-versus-unpersonalized GPT-4 contrast is +48.7% relative odds, 95% CI −2.9% to +127.6%; corrected P = .0678, rounded .07. No correction to the headline human-baseline effect was reported. These estimates are optional details, not numbers to memorize.
Optional unrelated binary-odds illustration · entirely fictional parcel collection. In one 100-customer group, 25 collect a parcel by the next day and 75 do not: probability 25/100 = 25%; odds 25/75 = 1/3. In a second 100-customer group, 40 collect and 60 do not: probability 40%; odds 40/60 = 2/3. The odds ratio is (2/3)/(1/3) = 2, a 100% increase in odds, while the collection probability rises by 15 percentage points and 40% collect in the second group. These are different quantities. The made-up binary outcome is not Salvi’s ordered agreement model; do not apply these counts or convert its reported relative-odds estimate into a percentage persuaded.
| Cell or contrast | Opponent / information access | Question answered | Population / analysis frame | Measured outcome / unit |
|---|---|---|---|---|
| H0 | Human without the recipient’s demographic profile | Unpersonalized-human reference condition | US adults in controlled assigned debates; see the sample/analysis detail | Ordered reported agreement; relative odds for reported contrasts, not a proportion changing view |
| H1 | Human with the recipient’s demographic profile | Personalized-human condition; access asymmetric | US adults in controlled debates; aggregate recipient is the participant without access to the opponent’s profile | Ordered reported agreement; relative odds for reported contrasts, not a proportion changing view |
| G0 | GPT-4 without the recipient’s demographic profile | Unpersonalized GPT-4 condition | US adults in controlled assigned debates; see the sample/analysis detail | Ordered reported agreement; relative odds for reported contrasts, not a proportion changing view |
| G1 | GPT-4 with the recipient’s demographic profile | Personalized GPT-4 condition | US adults in controlled assigned debates; see the sample/analysis detail | Ordered reported agreement; relative odds for reported contrasts, not a proportion changing view |
| G1 versus H0 | Opponent and profile access both change | Package contrast: +81.2% relative odds; 95% CI [+26.0%, +160.7%] | US adults in controlled assigned debates; see the sample/analysis detail | Ordered reported agreement; relative odds for reported contrasts, not a proportion changing view |
| G1 versus G0 | GPT-4 held fixed; profile access changes | Added-profile contrast: +48.7%; 95% CI [−2.9%, +127.6%]; corrected P=.0678 | US adults in controlled assigned debates; see the sample/analysis detail | Ordered reported agreement; relative odds for reported contrasts, not a proportion changing view |
How the researchers studied it
A fair test of a component changes that component while keeping the comparison intelligible. Random assignment reduces the chance that one condition simply receives people who were already easier to persuade. It does not make a timed experiment identical to an everyday conversation. For a binary event, odds are its probability divided by the probability of its complement; an odds ratio compares those odds across conditions. This paper uses an ordinal-regression analysis of ordered agreement responses. The optional binary-odds illustration below teaches the ratio idea only; it is not a explanation of that model or a conversion of its estimate into a head count.
Explain four opponent/information conditions: a human without the recipient’s demographic profile; a human with it; GPT-4 without it; and GPT-4 with it. In the human-personalized debates access was asymmetric; the aggregate analysis uses the recipient without access to the opponent’s profile. Identify which contrast keeps the opponent type fixed and which changes both opponent type and information access. Then say what outcome that contrast measured.
What they found
Personalized GPT-4 exceeded the unpersonalized-human baseline in the reported agreement outcome. That comparison establishes a difference between those conditions, not the added effect of demographic information alone.
The corrected direct GPT-4 comparison is compatible with several component effects, including no difference and a positive advantage. A privacy right need not depend on demonstrating a large persuasive benefit; that is a separate normative justification.
Where the claim stops
Paid adults defended assigned positions in a controlled task. The study did not observe election votes or establish an effect for every chatbot, population, or voluntary user.
A privacy rule can be justified by control over personal information even when its additional effect on persuasion is uncertain. Evidence about effectiveness and a judgment about rights answer related, different questions.
Experimental design; Aggregate results; Discussion; Human sample; and the linked Author Correction.
Read first: Start with the explanation on this screen. Then check what was compared, what was found, and what remains uncertain. The journal article link is optional.
Open-access updated publisher text and PDF. Read the 3 September 2026 correction: https://doi.org/10.1038/s41562-026-02588-0
Publisher article ↗Read the named author correction · 3 September 2026 ↗
