Large Language Models Augment or Substitute Human Experts in Idea Screening

Title: Large Language Models Augment or Substitute Human Experts in Idea Screening

Abstract: Firms that conduct crowdsourcing campaigns to generate ideas for advertising and product development often rely on an overwhelmingly manual process for screening the ideas. Internal experts rate thousands of ideas to identify a small set of promising ones that are then submitted for additional review. We provide estimates of the extent to which outputs from pre-trained AI models, namely large language models (LLMs), combined in a machine learning model trained on the evaluation data generated by the experts themselves and the clients’ final decisions, can augment or substitute for the effort of these internal experts. We use data from a crowdsourcing platform that employed experts to filter 74,436 ideas across 153 ideation contests organized on behalf of major advertisers. We show that evaluation work can be reduced by 28.4% relative to the status quo, of which 3.8% can be attributed to the LLM output and the remainder to the machine learning model appropriately reweighting and combining expert scores to match sponsor decisions. The use of the LLM can also render 5 out of 10 experts potentially redundant, in contrast to 3 if the machine learning model is used on its own. The experts whose scores are most substitutable with the LLM output are not necessarily the worst performers at the original task. Our results propose a concrete approach to integrate LLMs into the idea screening process and highlight the potential impact this may have on internal experts.