Events

Abstract: A growing literature uses large language models (LLMs) as synthetic participants to generate cost-effective and nearly instantaneous responses in social science experiments. However, there is limited guidance on when such simulations support valid inference about human behavior. This talk contrasts two strategies for obtaining valid estimates of causal effects and clarifies the assumptions under which each is suitable for exploratory versus confirmatory research. "Heuristic approaches" seek to establish that simulated and observed human behavior are interchangeable through prompt engineering, model fine-tuning, and other "repair strategies" designed to reduce LLM-induced inaccuracies. While useful for many exploratory tasks, heuristic approaches lack the formal statistical guarantees typically required for confirmatory research. In contrast, "statistical calibration" combines auxiliary human data with statistical adjustments to account for discrepancies between observed and simulated responses. Under explicit assumptions, statistical calibration preserves validity and provides more precise estimates of causal effects at lower cost than experiments that rely solely on human participants. Yet the potential of both approaches depends on how well LLMs approximate the relevant populations and on how effectively social scientists can tailor models to their purpose. The talk reviews practical challenges in simulating human behavior, outlines directions for future research, and discusses broader implications for the quantitative social sciences.

The Sociology Seminar Series for Trinity Term is convened by Ozan Aksoy and Zachary Parolin For more information about this or any of the seminars in the series, please contact sociology.secretary@nuffield.ox.ac.uk.