An updated meta-analysis of AI persuasion that separates attitudes from behaviour

Research question

Pooling randomized experiments published from 2023 to 2026, including those testing 2025–26 frontier models, how large is the persuasive effect of AI messages and conversations, relative to no message and to human persuaders, and how does it vary with model generation, conversation format, message length, topic, country and outcome type (attitude, vote intention, behaviour, persistence at follow-up)?

Why it matters

The only meta-analysis pooled seven studies up to May 2025 and found no average difference between AI and human persuaders, with large unexplained variation. Since then, several large preregistered experiments report frontier models beating expert humans, outperforming campaign ads and changing real behaviour. Readers currently have to weigh a dated null pooled estimate against individual newer studies. An updated synthesis that keeps attitude, behaviour and persistence outcomes apart would tell researchers and policymakers what the body of evidence now supports, and where it is still thin.

The debate lists this among what we still do not know: a newer meta-analysis of frontier-model persuasion

Evidence that motivates it

1. A 2025 meta-analysis of seven earlier studies found no average persuasiveness difference between LLMs and humans, with substantial heterogeneity. (LD10-HQ-C07 v1.0)
   A 2025 meta-analysis of seven studies found no average AI–human difference with substantial heterogeneity; an updated meta-analysis including 2025–26 frontier systems is named as what would change it.
   Cites: Hölbling, Maier & Feuerriegel, Scientific Reports 2025
2. In four preregistered 2025–26 UK experiments, frontier AI out-persuaded strong human text persuaders; its advantage over coached elite debaters disappeared when AI was constrained to the debaters’ message length and writing speed. (LD10-HQ-C08 v1.1)
   Newer UK experiments found frontier AI out-persuaded strong human text persuaders, with the advantage disappearing under human-like length and speed limits; message length and speed are therefore moderators a new synthesis should code.
   Cites: Hackenburg et al. 2026, arXiv:2606.16475
3. In two preregistered UK experiments, brief AI conversations increased real petition signing by 12.8 and 19.7 percentage points, while attitude effects did not predict behavioural effects. (LD10-HQ-C09 v1.0)
   Brief AI conversations increased real petition signing while attitude effects did not predict behavioural effects, which is why attitude and behaviour outcomes must be pooled separately.
   Cites: Hackenburg et al. 2026, arXiv:2604.09200

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Preregister a systematic review and meta-analysis (PROSPERO-style protocol on OSF). Search Web of Science, Scopus, ACM, arXiv, OSF, PsyArXiv, SocArXiv and SSRN for randomized experiments from 2023 to the search date in which AI-generated messages or conversations aim to change attitudes, intentions or behaviour. Code two contrasts separately (AI versus no-message control; AI versus human persuader), and outcome type (attitude, vote intention, behaviour, follow-up persistence). Code moderators: model and release date, interactive versus one-shot, message length, personalization, information-dense prompting, topic, country and language, sample source and preregistration. Fit multilevel random-effects models with robust variance estimation, test for small-study effects, and run sensitivity analyses that exclude studies under correction or expression of concern. Release the coded dataset as a living database that others can update.

Data needed

Published papers and preprints with effect sizes or raw data; replication files where available; two independent coders for screening and coding.

Feasibility

Master's thesis. Low: mainly researcher time; no data collection. A second coder is needed for reliability. Many recent studies are preprints and some have been corrected; record version and status for each included estimate. Outcomes are measured on different scales, so standardization choices must be preregistered.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 12 October 2026, checked again on 13 October 2026).

An earlier meta-analysis (Huang & Wang, 2023) pooled 121 experiments comparing AI and human communicators and separated perceptions, attitudes, intentions and behaviour, but mostly predates LLM-era studies. The only LLM-specific meta-analysis found (Hölbling et al.) closed its search on 22 May 2025, included seven studies and pooled only the AI-versus-human contrast. Large later experiments (Hackenburg et al., June 2026; Chen, Kalla & Le, March 2026) report primary results but no synthesis, and a study of nine automated persuasion benchmarks (Dementaviciute et al., October 2026) compares methods, not pooled human-subject effects. No meta-analysis covering 2025-26 frontier-model experiments, separating AI-versus-control from AI-versus-human contrasts, or separating attitude, behaviour and persistence outcomes for LLM persuasion was found.

Literature check. No meta-analysis pooling 2023-2026 LLM persuasion experiments, with separate AI-versus-control and AI-versus-human contrasts and separate attitude, behaviour and persistence outcomes, was found. Two syntheses bound the space: Hölbling, Maier & Feuerriegel (Scientific Reports 2025; arXiv:2512.01431) pool seven LLM-versus-human studies with a search closed in May 2025, and Huang & Wang (Journal of Communication 2023) pool 121 earlier experiments comparing AI and human agents and do separate perceptions, attitudes, intentions and behaviours, but predate most LLM-era studies and only cover the AI-versus-human contrast. The verdict moves from OPEN to PARTLY_ADDRESSED because outcome-type separation has been done for pre-LLM AI communicators. Checked by Claude (SocioLens Project Lead) with AI research agents, by web search; not a person and not a domain expert.

Where the search could not reach. Screened by web search on the date shown. PROSPERO, OSF registrations, Google Scholar and SSRN were not searched directly, so a registered but unpublished review could exist. A September 2025 ResearchGate entry titled "The Persuasive Power of LLMs: A Systematic Literature Review and Meta-Analysis" could not be opened; it may be an earlier version of Hölbling et al. or a separate review. Absence of a hit is not proof that the work does not exist. Literature check of 2026-10-13: PROSPERO was not searched directly and an OSF/PsyArXiv record (osf.io/preprints/psyarxiv/ehg7n) surfaced by the protocol query could not be read, so a registered but unpublished review may exist. The Current Opinion in Psychology review 'Using conversational AI to reduce science skepticism' (ScienceDirect S2352250X25002295) could not be opened; it appears to be a short review rather than a meta-analysis. Google Scholar was not searched. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Hölbling, Maier & Feuerriegel, "A meta-analysis of the persuasive power of large language models," Scientific Reports 15, 43818 (12 Dec 2025)
  Seven studies, 17,422 participants, 12 effect sizes, search closed 22 May 2025; pooled Hedges' g = 0.02 for AI versus humans with I² of about 76%. Models up to GPT-4o and Claude 3.5 and 3.7 Sonnet. As described, it pools the AI-versus-human contrast only and does not report behavioural or persistence outcomes separately.
• Hackenburg, Wagner, Hewitt, Tappin, Saunders, Kirk, Margetts & Summerfield, "AI systems out-persuade expert humans," arXiv:2606.16475 (15 Jun 2026)
  Four preregistered experiments (18,978 conversations, 6,923 people) comparing AI with laypeople, tournament winners, canvassers and champion debaters. A primary study a new meta-analysis would include, not a synthesis.
• Chen, Kalla & Le, "Benchmarking Political Persuasion Risks Across Frontier Large Language Models," arXiv:2603.09884 (10 Mar 2026)
  Two survey experiments (19,145 Prolific participants, US policy issues) testing multi-turn conversations with seven frontier models against standard campaign ads; reports the models outperform the ads, with persuasiveness varying by model. Primary study, not a synthesis.
• Dementaviciute, Vaitonyte & De Bie, "LLM Persuasion Is in the Eye of the Evaluation," arXiv:2610.10232 (7 Oct 2026)
  Runs nine published automated persuasion-evaluation methods on fifteen models and finds weak agreement between them. Compares automated benchmarks, not human-subject experiments, so it is not a meta-analysis of persuasion effects.
• Huang & Wang, "Is artificial intelligence more persuasive than humans? A meta-analysis," Journal of Communication 73(6), 552-562 (2023) Judged from an abstract, summary or search listing only
  Meta-analysis of 121 randomized experiments (N = 53,977) comparing AI and human agents: overall equally persuasive; AI less effective on behavioural intentions, no significant difference on perceptions, attitudes or actual behaviours; moderators include AI role, communication direction, setting and demographics. Separates outcome types, but pools AI-versus-human contrasts only and mostly predates LLM-era experiments (search dates not stated on the page). Read from the HKBU repository record.
• Hölbling, Maier & Feuerriegel, "A Meta-Analysis of the Persuasive Power of Large Language Models," arXiv:2512.01431 (Dec 2025)
  Preprint version of the Scientific Reports article already listed. No later update or extension citing it was found in this round.
• Hackenburg, Hewitt, Wagner, Tappin & Summerfield, "Artificial intelligence can persuade people to take political actions," arXiv:2604.09200 (2026) Judged from an abstract, summary or search listing only
  Authors, title and date (10 April 2026) checked on the arXiv page; the paper itself was not read. From its title, a primary study with a behavioural (political action) outcome that a new meta-analysis would include; not a synthesis.

References

Sources behind the motivating claims:

• Hölbling, Maier & Feuerriegel, Scientific Reports 2025 (SRC-LD10-HOLBLING-MAIER-FEUERRIEGEL-2025, original)
• Hackenburg et al. 2026, arXiv:2606.16475 (SRC-LD10-HACKENBURG-ARXIV-2606-16475, original)
• Hackenburg et al. 2026, arXiv:2604.09200 (SRC-LD10-HACKENBURG-2026, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld10_06_v2,
  author = {{SocioLens project}},
  title = {An updated meta-analysis of AI persuasion that separates attitudes from behaviour},
  year = {2026},
  date = {2026-10-13},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld10-06/v2/}},
  note = {Research opportunity RO-LD10-06, version v2}
}

@misc{ro_ld10_06_nearest_1,
  title = {Hölbling, Maier \& Feuerriegel, "A meta-analysis of the persuasive power of large language models," Scientific Reports 15, 43818 (12 Dec 2025)},
  url = {https://www.nature.com/articles/s41598-025-30783-y},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld10_06_nearest_2,
  title = {Hackenburg, Wagner, Hewitt, Tappin, Saunders, Kirk, Margetts \& Summerfield, "AI systems out-persuade expert humans," arXiv:2606.16475 (15 Jun 2026)},
  url = {https://arxiv.org/abs/2606.16475},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld10_06_nearest_3,
  title = {Chen, Kalla \& Le, "Benchmarking Political Persuasion Risks Across Frontier Large Language Models," arXiv:2603.09884 (10 Mar 2026)},
  url = {https://arxiv.org/abs/2603.09884},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld10_06_nearest_4,
  title = {Dementaviciute, Vaitonyte \& De Bie, "LLM Persuasion Is in the Eye of the Evaluation," arXiv:2610.10232 (7 Oct 2026)},
  url = {https://arxiv.org/abs/2610.10232},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld10_06_nearest_5,
  title = {Huang \& Wang, "Is artificial intelligence more persuasive than humans? A meta-analysis," Journal of Communication 73(6), 552-562 (2023)},
  url = {https://scholars.hkbu.edu.hk/en/publications/is-artificial-intelligence-more-persuasive-than-humans-a-meta-ana-2/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld10_06_nearest_6,
  title = {Hölbling, Maier \& Feuerriegel, "A Meta-Analysis of the Persuasive Power of Large Language Models," arXiv:2512.01431 (Dec 2025)},
  url = {https://arxiv.org/abs/2512.01431},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld10_06_nearest_7,
  title = {Hackenburg, Hewitt, Wagner, Tappin \& Summerfield, "Artificial intelligence can persuade people to take political actions," arXiv:2604.09200 (2026)},
  url = {https://arxiv.org/abs/2604.09200},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_ld10_holbling_maier_feuerriegel_2025,
  title = {Hölbling, Maier \& Feuerriegel, Scientific Reports 2025},
  url = {https://doi.org/10.1038/s41598-025-30783-y},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld10_hackenburg_arxiv_2606_16475,
  title = {Hackenburg et al. 2026, arXiv:2606.16475},
  url = {https://arxiv.org/abs/2606.16475},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_ld10_hackenburg_2026,
  title = {Hackenburg et al. 2026, arXiv:2604.09200},
  url = {https://arxiv.org/abs/2604.09200},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}