Artificial Agents in Operations Management Experiments
Built on large language models (LLMs), silicon samples offer a promising approach for advancing behavioral research by simulating decision-making processes and replicating human decision-making in controlled experiments. This study evaluates LLMs’ potential to replicate behavioral hypotheses in operations management (OM) by conducting nine studies from the Management Science Replication Project. I develop an accessible framework where GPT-4o agents are assigned roles based on System 1 decision-making and participate in the experiments. Remarkably, the silicon samples replicated the core hypotheses in eight of the nine studies, with treatment effects closely matching the direction and magnitude of human decisions. The natural language explanations generated by GPT agents offer insight into their decision logic, revealing whether observed choices align with the behavioral theories being tested. These findings position silicon samples as a valuable tool for replication research, helping researchers distinguish between sampling limitations and theoretical divergence. More broadly, the results support using silicon samples to scale replication, design interventions, and conduct high-fidelity pretests before lab or field deployment.
Bio: Sam Kirshner is an Associate Professor and Head of School of Information Systems and Technology Management at the UNSW Business School. Sam completed his PhD in Management Science at Queen’s University in Kingston, Canada. His primary research interests lie in analysing behavioural decision-making in operations and technology management and studying how algorithms and artificial intelligence impact and and make decisions. His research and commentaries have been published in prestigious academic journals, including Management Science, Production and Operations Management, Manufacturing & Service Operations Management, Decision Sciences, the European Journal of Operational Research, Journal of Behavioral Decision Making, and Science, and he is a Regional Editor for the Journal of Supply Chain Management and is an Associate Editor at Decision Sciences. He is also the co-author of two books, a textbook entitled Business Analytics: A Management Approach and a book on responsible AI called Checkmate Humanity: The How and Why of Responsible AI.
Online profile: https://www.unsw.edu.au/staff/sam-kirshner