Artificial Agents in Operations Management Experiments
Built on large language models (LLMs), silicon samples offer a promising approach for advancing behavioral research by simulating decision-making processes and replicating human decision-making in controlled experiments. This study evaluates LLMs’ potential to replicate behavioral hypotheses in operations management (OM) by conducting nine studies from the<Management Science Replication Project. I develop an accessible framework where GPT-4o agents are assigned roles based on System 1 decision-making and participate in the experiments. Remarkably, the silicon samples replicated the core hypotheses in eight of the nine studies, with treatment effects closely matching the direction and magnitude of human decisions. Crucially, the natural language explanations generated by GPT agents offer insight into their decision logic, revealing whether observed choices align with the behavioral theories being tested. I also provide strong evidence that the agents’ decisions are shaped by task structure and feedback across rounds, supporting an interpretation of in-situ reasoning rather than recall and reproduction of published results. These findings position silicon samples as a methodological tool that complements existing experimental, analytical, and simulation approaches in OM, while contributing evidence to broader debates on the use of LLM-based agents as research subjects. For researchers, silicon samples support scalable replication, diagnostic analysis of decision processes, and early-stage intervention testing. For decision makers, the approach enables rapid exploration of behavioral responses across complex operational settings prior to costly deployment.
Sam Kirshner
Sam Kirshner is an Associate Professor in the School of Information Systems and Technology Management at the University of New South Wales. Sam completed his PhD in Management Science at Queen’s University, in Kingston, Canada. His primary research interests lie in analyzing behavioural decision making in operations and technology management and studying how algorithms and artificial intelligence impact decision making. His research and commentaries have been published in prestigious academic journals including Management Science, Decision Sciences, Production and Operations Management, Manufacturing & Service Operations Management, the European Journal of Operational Research, Journal of Behavioral Decision Making, Tourism Management, and Science. Sam is also active in program development and teaching in Business Analytics at the UNSW Sydney. Sam teaches data visualisation, predictive analytics, and AI ethics to undergraduates, postgraduates, and MBAs. He is also the co-author of a new textbook entitled Business Analytics: A Management Approach and a member of the Ethical AI Advisory.