Generative AI has empowered marketing firms to create huge volumes of creative assets, from promotional emails to original infographics. Today’s marketers can generate complete content plans, including relevant text and images, completely from scratch.
While firms rush to implement this ever-evolving technology, practical questions loom. For example, being able to generate any kind of content with a simple prompt is only useful to the firms that know what kind of content will accomplish their ultimate objectives.
According to James Reeder, assistant professor in the Department of Marketing at the University of Tennessee, Knoxville, Haslam College of Business, the possibilities of AI have raised pragmatic questions.
“We have this explosion of generative AI, especially in marketing, and firms are grappling with how to actually use it,” says Reeder. “That’s the big question right now.”
Beyond A/B Testing
Reeder’s new paper, “Evaluating Novel Unstructured Treatments with Generative AI: A Causal Prediction Framework,” may provide some clarity. In it, he and coauthors demonstrate how firms can use data from past campaigns to anticipate how AI-generated content will affect real marketing outcomes.
According to Alex Zablah, head of Haslam’s marketing department and the Gerber/Taylor Distinguished Professor and Kinney Family Faculty Research Fellow, Reeder and co.’s research takes a thoughtful, balanced look at AI’s potential.
“James’ research doesn’t treat AI as the ultimate answer,” Zablah notes. “It correctly frames AI as one input into a decision that a marketer must ultimately make. The paper links a firm’s own past results to the new content AI produces, which offers the tool the appropriate context for supporting decision-making.”
The team’s research could give marketers a smarter alternative to traditional A/B testing, which involves creating two different versions of an asset — like an ad or email — or campaign, then sharing both versions to part of their audience to see which performs better.
While A/B testing can provide useful results, it also means some customers inevitable receive marketing materials that may not work well.
Reeder’s framework takes a more proactive approach. By drawing on historical data, it can help marketers predict how a message will perform before customers ever see it. Marketers can evaluate new marketing assets based on how similar they are to previous ones and use those insights to make more informed decisions.
“The practical takeaway for marketers is clear,” says Zablah. “If a new message looks like things you’ve run before, your own data can very quickly tell you a lot about how it will perform, allowing you to get content into the marketplace faster. If it’s truly new, the model tells you that too and points you to a need for testing.”
Working within Limits
Reeder notes that the predictive power of AI is at its best for messaging that is like what a company has tried before. The more marketers innovate with new brand positioning or sentiment, the more uncertainty they encounter.
This uncertainty is critical to Reeder’s model. In his approach, AI doesn’t tell managers what message to use. Instead, it shows how confident they can be in a prediction, giving them better information to guide their next move. Managers still make the final decision to stick to familiar messaging or try something new.
Reeder’s research also highlights generative AI’s limitations, noting that marketing assets chosen entirely by generative AI were predicted to perform significantly worse than AI-generated options reviewed and ranked using the company’s past campaign results. In other words, human input is necessary to effectively interpret and implement AI-generated data.
“Firms should not blindly adopt what ChatGPT says to do,” he says. “The outcome you get from generative AI is only as good as how well you immerse it in the information from your firm.”
Making Informed and Effective Decisions
Reeder points to one example that explains why it’s important to understand a prediction’s limit. One retailer examined in his research sent 34 very similar promotional emails. Only one email, a birthday greeting, stood apart from the rest. Because it was so different from the retailer’s previous message, the AI framework predicted its performance as unreliable, even though this birthday message ultimately had the strongest effect on customer spending.
According to Reeder, this example illustrates the usefulness of AI as a predictive tool, while also emphasizing the ongoing importance of human judgment.
“Human oversight is more critical now than ever,” he summarizes. “You need grounding in firm-level outcomes or human judgment to say ‘that’s a good idea’ or ‘that’s a bad idea.’”
“Evaluating Novel Unstructured Treatments with Generative AI: A Causal Prediction Framework,”cowritten by Reeder, Paul B. Ellickson with the University of Rochester’s Simon Business School, Wreetabrata Kar with the Smeal College of Business at Penn State; and Guang Zeng with the Nova School of Business and Economics in Lisbon, Portugal, was published by the Journal of Marketing Research.
Contact
Related News
UT Entrepreneurship Center Alumni Shine at Knoxville Entrepreneur Conference
Entrepreneurs from across Tennessee gathered in Knoxville for 865Fest on September 29, bringing together founders, students and members of the...
Read ArticleUT Student Entrepreneurs Turn Business Ideas into Funding with Entrepreneurship Center
Students competed in a pitch competition for $3,000 in Roe Prize funding.
Read ArticleUT’s Haslam College of Business Sets Record for Total Undergraduate Enrollment
The college achieved an all-time high in overall undergraduate enrollment while tying its highest first-year student retention rate.
Read ArticleUT Haslam Marketing Professor Honored by Eminent International Body
Garriy Shteynberg, who previously taught in UT’s psychology department, was named a fellow by the Society for Personality and Social...
Read Article