Preview: not announced. Search engines asked not to index.

Proposal pack · RO-LD08-03 v1

Do developers' usage statistics hold up when someone else classifies the conversations?

To keep this as a PDF or print it, use your browser's Print command (Ctrl+P, or Cmd+P on a Mac) and choose "Save as PDF". Links print with their addresses. This pack gathers what the opportunity page already holds; nothing in it is new.

Read this first. This is a research hypothesis drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-08 debate on 16 October 2026, not a finding. Not yet reviewed. Every check so far was done by AI; no person or domain expert has checked it yet. The literature screen below was done by AI web search and can miss relevant work. Version v1, permanent address https://sociolens-web.pages.dev/opportunities/ro-ld08-03/v1/.

Instrument stability One PhD chapter or paper Needs a budget (medium or higher) Partly addressed

Research question

When the same chatbot conversations are classified independently of the developer, with a validated scheme and human-coded benchmarks, how far do headline usage measures such as the automation share, the work share and task success move away from the figures developers publish?

Why it matters

Usage statistics from AI developers are now cited as evidence about how people use AI, but each is produced by the developer's own model and checked in-house. One developer's automation share moved from 49% to 45% when a different model classified the same data. An independent study of public logs borrowed developer-linked classifier prompts without validating them, so independent data alone do not supply an independent instrument. Without an independent instrument it is unknown how much of the published picture is the instrument.

The debate lists this among what we still do not know: How stable are vendor usage measures when the classifier, the validation and the data owner change?

Evidence that motivates it

  1. In its September 2025 Economic Index report, Anthropic reported that the share of automation in Claude.ai conversations depended in part on which model did the classifying: re-running the same August 2025 data with Claude Sonnet 3.7 instead of Claude Sonnet 4 gave 45% automation instead of 49%, though the rise in directive use remained. (LD08-SL-C18 v1.0)
    The automation share moved from 49% to 45% when another model classified the same conversations.
    Cites: Anthropic (2025), “Anthropic Economic Index report…”
  2. Anthropic’s first Economic Index report classified Claude conversations as 57% augmentation and 43% automation; the classifications were made by Claude and checked by the authors on 150 hand-validated conversations drawn from those for which users had submitted feedback, with 90.7% agreement on the augmentation–automation label and 86% to 95.3% acceptable task assignments by level of detail. (LD08-SL-C17 v1.0)
    Classifications made by the developer's model and checked on 150 conversations chosen from those with user feedback.
    Cites: Handa et al. (2025), “Which Economic Tasks are Performed…”
  3. Anthropic’s January 2026 Economic Index report added measures such as “task success,” which “measures Claude’s assessment of whether Claude completes tasks successfully”; for Claude.ai the classifiers were checked against a human researcher on a small set of transcripts for which users had given feedback, and the report calls them “directionally accurate”; it found augmentation rising to 52% of Claude.ai conversations. (LD08-SL-C19 v1.0)
    Task success is the model's own assessment of whether it completed the task.
    Cites: Anthropic (2026), “Anthropic Economic Index report…”
  4. Analysing 37,845 Australian conversations from WildChat, a public dataset of ChatGPT logs collected outside OpenAI, university researchers classed 61.1% as work-related against 37.3% in OpenAI’s published global figures; to classify them they adopted the classifier prompts and categories of OpenAI-co-authored work. (LD08-SL-C22 v1.0)
    An independent study of public logs reused the developer-linked classifier prompts without a validation of its own.
    Cites: Ma et al. (2026), “How People Use ChatGPT in Australia: A…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Take a public or donated conversation corpus, build a human-coded benchmark sample stratified by topic and language, and classify the corpus with several models and prompt sets, including the prompts developers publish. Report each headline measure with its spread across classifiers and its agreement with the human benchmark, and compare with the developers' published figures for the same period.

Data needed

A licensed public or donated conversation corpus; published developer taxonomies and prompts; trained human coders; access to several classifier models.

Feasibility

One PhD chapter or paper. Medium; human coding of the benchmark sample and model inference costs. Public and donated corpora differ from the developers' user bases, so the study measures instrument sensitivity, not the true population share.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 16 October 2026; not yet checked in depth).

The developer's own model-swap check and one independent study of public logs show that the measures move, but no study was found that classifies the same corpus with several instruments against a human benchmark and compares the result with developers' published figures.

Where the search could not reach. Screened by web searches and an arXiv record lookup on the date shown; an R package that re-reads the published Economic Index aggregates appeared in search results but was not opened. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

References

Sources behind the motivating claims:

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld08_03_v1,
  author = {{SocioLens project}},
  title = {Do developers' usage statistics hold up when someone else classifies the conversations?},
  year = {2026},
  date = {2026-10-16},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld08-03/v1/}},
  note = {Research opportunity RO-LD08-03, version v1}
}

@misc{ro_ld08_03_nearest_1,
  title = {Anthropic, Economic Index report, September 2025 (LD08-SL-C18)},
  url = {https://www.anthropic.com/research/anthropic-economic-index-september-2025-report},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld08_03_nearest_2,
  title = {WildChat study of Australian conversations (LD08-SL-C22)},
  url = {https://arxiv.org/abs/2609.28990},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld08_03_nearest_3,
  title = {Yan, Nguyen, Su, Lieffers \& Le, "ShareChat: A Dataset of Chatbot Conversations in the Wild," arXiv:2512.17843 (2025)},
  url = {https://arxiv.org/abs/2512.17843},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_anthropic_economic_index_2025_09_15,
  title = {Anthropic, “Anthropic Economic Index report: Uneven geographic and enterprise AI adoption,” 15 September 2025},
  url = {https://www.anthropic.com/research/anthropic-economic-index-september-2025-report},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_handa_et_al_arxiv_2503_04761,
  title = {Handa, Tamkin, McCain, Huang, Durmus et al., “Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations,” arXiv:2503.04761, 2025},
  url = {https://arxiv.org/abs/2503.04761},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_anthropic_economic_index_2026_01_15,
  title = {Anthropic, “Anthropic Economic Index report: Economic primitives,” 15 January 2026},
  url = {https://www.anthropic.com/research/anthropic-economic-index-january-2026-report},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_ma_et_al_wildchat_australia_arxiv_2609_28990,
  title = {Ma, Gero, Canonne, Jin \& Thilakarathna, “How People Use ChatGPT in Australia: A WildChat Analysis,” arXiv:2609.28990, 2026},
  url = {https://arxiv.org/abs/2609.28990},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

Back to the opportunity · All research opportunities