How In-market Testing Delivers Richer Consumer Insights

Priscilla McKinney: Hello and welcome to Ponderings from the Perch, the Little Bird Marketing Company podcast. I'm Priscilla McKinney with you as always, CEO and mama bird over here. And this is the podcast where I invite my friends in marketing and market research to come over and talk about what they're actually doing. We really set aside all of the BS and we just have good conversations about just how hard our jobs are sometimes.

I love reaching out to friends who just can get on and are so unconsciously competent about their world that they can easily impart expertise to you and give you maybe a new way of thinking about your product or your service or what it is to really be an insights professional today. So you're going to love my guest today. I have Rachel Buss. She is the vice president of strategic insights at Curion and she is a leading voice in how consumer behavior is reshaping the world and really with all of her years of experience, you know, she has honed in on the age old dilemma we're going to start with today, which is the say do gap. So Rachel, welcome to the show.

Rachel Buss: Yeah, thanks for inviting me. It's been excited to be here.

Priscilla McKinney: Yeah, well, it's fun just talking with you. Honestly, we shoot the shit a lot about what's going on in the industry. And I find that your perspective with all of your years of experience with controlled testing and then the other side of it, which is real world performance of products and services. I love how you can so easily talk about how data actually turns into things the team is going to do in order to make good.

And in order to really service the customer better, I think that's one thing I really love about your approach. There are so much we could talk about because you guys work in QSR, you work in CPG, you work in most interesting, like recently I heard you guys talking at one of the conferences about GLP-1 users and how things are changing with them. That was just totally riveting. But I love how you say yeah, that's right now and that's current but you still bring a lot of academic rigor to the study. So we're gonna start with this say-do gap kind of the crazy frustration that always is market research. So tell me about where you start there and where you're hearing mounting frustration from clients.

What are they not knowing about what's going on with consumers as opposed to what they think they're doing and what they're actually doing?

Rachel Buss: Yeah, I think the say-do gap, I mean, it's a catchphrase in our industry, right? But it's been around as long as I have been in the industry, which is about 15 plus years now. Everyone wants to be able to predict what's coming next. And the only way we can really try to do that or get semi-close is based off of history and try to, you know, forecast and cast our crystal balls a little bit to see, you know, what's coming based off of what's been.

Unfortunately, consumers aren't super reliant on remembering what they did, what they said, what they bought. Can you remember what you had for breakfast this morning? Do you remember what you had for dinner last night? Probably not. Do you remember what was in your grocery cart on your last shopping trip or how many times you went shopping? Probably not. So what we find with a lot of these different issues is not that consumers are lying. It's just that they don't remember.

So they're trying, you know, their best guess, but also, you know, just like anyone sits there and tries to guess the number of jelly beans in a jar. How many times did I buy this product? I don't know, four, five, six, maybe. So things like that, you know, makes all of us a little bit of an unreliable narrator in our own lives. And we just kind of struggle on that memory piece. So that means our forecast, our predictions, our crystal balls, can often be off because what consumers say they did is not what they actually did. So what they think they will do is most likely not what they will actually do going into the future.

Priscilla McKinney: I love that and you put that so succinctly, but I got to tell you a funny story to this. So I had Raina from StarKist on the show and I was talking with her about, you know, how much I love their product, their pouches that are like eat and go, you know, and I was talking to her and like, and I am not usually the person doing any of the grocery shopping. I haven't been in a grocery store more than 10 times in the last 20 years, right?

And so I said to her, I'm like, yeah. And I, my husband was gone. So I went into Target and I got this StarKist pouch and then they had other ones. And then there was a new sriracha flavor and blah, blah, blah. And we're talking about it. And then after I finished recording the podcast episode, I went in my kitchen and I went to eat my lunch. And yes, there was a StarKist pouch, but the sriracha flavor one I was talking about was a different brand.

And I tried it out on a whim and I was the unreliable narrator. I was like, I didn't even tell her the truth and I didn't mean it. And so this is so real. Everybody feels that. And then we're horrible also at predicting what we're going to do. I'm going to run into the store and get these two things. And no, we're not. That's not what we're going to do a lot. So when you have clients coming to you with this and saying, we're trying to make plans and grow our business based on data that consumers are reporting to us and trying to figure out what are they likely to do, tell me what those conversations sound like, how do you kind of ground it and get started with them?

Rachel Buss: Yeah, when you first kind of start with these conversations with clients, you start out with this idea of what do you think you know? So you do this kind of background into the data that they have and you understand where that data came from. Is this from an omnibus study? Is this from central location tests in the lab? Is this, did you go in a consumer's home? Like what type of data do you have?

We're also now hearing from clients that they have synthetic data from synthetic panelists that they're pulling into this data set or second party, third party data. All of these different kinds of sources are funneling into this. What do I think I know today based off of that and getting an idea, you can really start to see how much time they have spent literally face to face with their consumers. We all have these giant segmentation studies. We all have these profiles of these people we name Barb or Bob and they're a certain age and they do these hobbies and they have this many kids and it's necessary right because you got to have some targets and you have to be general.

But you're not necessarily having the full picture, the full color, unless you have spent time learning face to face with a consumer and not necessarily literally face to face but on a video, in an ethnography, shopping along with them, any of those kinds of elements can really help you add additional color to the type of data you already have so that you can better inform and understand how big your say-do gap actually is.

Priscilla McKinney: Right. Tackling the say-do gap, I mean, I'm sure it's different with everybody. But so let's talk about that because from your vantage point, you can tackle that say-do gap a couple of different ways. And there is a time and place, I'm sure, for that centralized testing and in a little more controlled environment. But then you already mentioned shop-alongs and seeing what's going on. There's like, you know, what are consumers doing out in the wild? So what's the range if someone's not really familiar with some of the options?

Rachel Buss: Yeah, so there can be any form, big size, small size. So if you want to go small data size, you're talking about a qualitative research study. That might be your ethnographies, which traditionally infer that you are side by side or a shop along, but you know, we're digital now, so we do a lot of things that our team calls self-knows, which are self ethnographies where we're following a lot of things through online journals, video diaries, those kinds of data types of capture.

And then you can go all the way as far as to quantitative studies, but you're still sending the consumer to go and do a thing and follow that journey along with them so that they've gone and bought the product or they've gone to that store that you wanted, guaranteed based off of the receipt that they send you a picture of, that yes, they bought the product. Yes, they poured out as much as you wanted them to do. Because, you know, people aren't necessarily, again, the reliable narrators in their world. And without some of that validation, whether that is physical forms of receipts or pictures or things like that, maybe they did, maybe they didn't do the things that you asked them to do.

Priscilla McKinney: Right. Right. Right. Where, how, how far skewed does this get? And I don't want to have a gotcha moment. Like what's the worst thing you've seen here, but tell me about how you've seen how important that context is, that they did go to that particular store or they did pick that particular product or they, this was a, you know, a beauty product they tried or, you know, et cetera. So tell me a little bit about what you've seen in the context.

Rachel Buss: So one of my favorite examples, in one of our carry-on intelligence reports, we sent consumers to restaurants, to casual dining restaurants, to experience a burger. We wanted to understand the full in-store and restaurant experience and all of those kinds of things.

By doing this, or because we wanted to be sure, we asked for photo verification of both the receipt as well as the burger that they received. And as we were going through the data, what we realized we failed to specify in our recruit is that we wanted US restaurant consumers. What we ended up with was two people who, I can only assume are military, my guess, but they went to a chain restaurant that we were interested in in the UAE, which was fascinating information, but their experience was totally different than those from the US.

They still were able to order the product. We still compensated for their time. They didn't do anything wrong. But the context of where they were and the competitive set for them was very different than the people here in the US and their point of views and competitive sets. So it was quite fascinating. We actually had to Google. We're like, is this restaurant in this country? We had no idea. So we spent a lot of time on Google and we're like, it's legit. Like, it's not AI. They really did the thing. They did the task. So it was quite a learning curve for us in that some restaurants are everywhere and you really got to be specific about what you want them to do and where you want them to do it.

Priscilla McKinney: Yeah, okay. That's interesting. So let's say we even get that right, let's ask a little bit more about the rigor because one concern that researchers often raise about real world approaches is the variability. Okay, let's take UAE out. But what about the environment like, you know, some people say well, we want to test in this controlled environment. But how do you defend the data validity from one to the other. What are you thinking about in terms of rigor inside the lab and outside?

Rachel Buss: Yeah, so there's a time and a place for both. Inside the lab, very controlled is when you're still tinkering. It's still not right or you want to test against your competitor if you want to make sure that everything is as standardized as possible so everything has an even playing field.

However, when you're ready to go to launch or when something is already in market, you have to realize that the world is messy. The world is chaotic. The reasons and the way you interact with a product are different in the lab versus when your child is screaming at you and you're trying to give them their tablet so you can make dinner and get it on the table. Those contexts are totally different and it changes how you interact with it in such a way that the data itself can be very different, which isn't good or bad. It's just different. Your objective and where you are at in your total product lifecycle impacts what type of research you should potentially do.

Priscilla McKinney: Yeah. You know, I find that interesting that, you know, they test in the lab, let's say everything goes well, and then it goes into market. How often do you see that things that looked like they were going to be good then when they got in the real world, to your point being messy and chaotic, that it didn't test well or it's not performing well? Like, you know, how often do you see that?

Rachel Buss: Yeah, it actually happens a lot. And it is sometimes about because when you test something in a lab, maybe you're just seeing like exactly how much strawberry flavor I should be adding to this yogurt. So maybe you're just trying to fine tune, dial in and not really trying to infer, predict how well something is doing. But in my restaurant world, a lot of time we're testing what the next great sandwich is and it performs great in the lab, but then once they launch it and they lose control, it doesn't do as well.

And what we're also finding with that kind of inference is when it's made in the controlled setting, it's controlled. Perfect. By the chef, by the head chef, by all of these culinary people. One person is putting on the whipped topping. One person is putting on a cherry. One person is putting on a lid versus a poor college student in lunch hour rush who is trying to run the register, take the orders, as well as do all of the food assembly. So they're probably not taking as much tender care and treating each shake like it's their baby like you might be doing in a central location test or any sort of in the lab testing. So once you lose control.

Priscilla McKinney: Right. And who's to say that poor college student actually got training on this new product? They're like, I haven't seen this sandwich. What's going on?

Rachel Buss: Maybe they did, maybe they didn't, maybe they weren't paying attention, maybe they were texting their friend while it was happening, you know, a lot of those things can happen, but even in a grocery store type sense or dollar discount, CVS, drug stores. The context is also different because when it's in a central location test, when it's in a lab, you're focusing on this one type of product.

You're usually in a like a white walled kind of space where you're like zeroed in on this. You don't necessarily always have the packaging. You don't always have the price. You don't always have the competitor set. You don't necessarily have all the claims that competitors are making that are also trying to be loud and annoying and drive your attention away from the one product you may have walked in with the intent to purchase.

So if you don't have all that additional context, you know, when you're at the end of the life cycle of the product there may be things that you miss and frequently we do because not everyone goes that far. And then so in my past life I worked at a company that technically no longer exists, Kellogg's. It has been split apart and sold and absorbed by other pieces now. But when I worked in research there, what happened is that a product would do well in the lab, it would launch, and then they would blame the product when it failed.

They weren't necessarily blaming the marketing campaigns, investment spend, what sales everybody else was doing at the time. They just always assumed something was wrong with the product itself rather than really understanding that it could have been an amazing product, but maybe it was bad timing. Maybe somebody else had something better. Like there's all of these different complex factors that go into product performance.

Priscilla McKinney: Yeah, yeah, I mean it is complicated and things are coming fast and also if you look at any shelf it's so competitive. Oh my gosh. It's like things are just vying for your attention and you really do have a split second in there. So you gave me a surprise which was about the burger, doing this kind of testing for QSR. I'm kind of curious if there are other surprises, but my question is a little bit different. Are there surprises you think in certain verticals like do more changes happen with QSR or with beauty or with food or, you know, have you seen any kind of trends like that?

Rachel Buss: Yeah, I think a lot of the changes at the point of sale, I think the biggest changes are in something that is sold in a grocery store. I think the biggest changes though in something of like the product performance, typically I see in a restaurant type setting because of who's made it now. And the differentiation of restaurant one versus restaurant two, different people prepared that product versus you go into the grocery store, you're running massive lots on production runs where they're very standardized on exactly your moisture specs and your texture things and all my scientist friends are gonna hate me for forgetting all of the words, but they're very standardized and controlled for very specific reasons, because they want those consistent experiences.

And restaurants strive for that. McDonald's came out with, why McDonald's has been so successful is that a burger is a burger, and in theory, it's a very similar burger across those experiences, no matter where you go to get it. But a lot of those things just have inherent human variability that you can't control for.

Priscilla McKinney: Yeah, yeah. Yeah, for sure. And you know, it's like, I'm going to go out on a limb here. I hope nobody at Starbucks hates me. Lord knows I spent enough money there, so they should really love me. But you know, I don't love their coffee the best, but it is the most consistent, right? And so there are these kinds of feelings across, you know, experiences really when things get into market.

And there are companies, some of them are longer existing clients, but not always, who really have honed in on the a burger is a burger is a burger is a burger, you know, and that's interesting. And I imagine that brands know if they're one of those brands or not, but maybe that's why they're coming to you if they feel like they aren't. But all of this is super interesting, but it really all comes down to what it is you deliver in the end. Businesses are coming to you because they want the data, they want the clues, they want the understanding in order to just make a decision today. So what does it look like for you once you've done all this work with your teams globally and you put it all together? What does it sound like when you're with the client and you're hoping this will actually change something, whether in formulation or packaging decision or a new go-to-market strategy or things like that? What are those conversations sound like?

Rachel Buss: Yeah, so those conversations differ based off of what the hypothesis was. Quite honestly, in our field, it's a lot easier to have those conversations if their hypothesis was confirmed. If they're like, we thought this and we're like, yes, exactly, you're on track, you're on right, your strategy is great, you know, go forward and prosper and you guys are perfect. The harder conversations is when the data is saying something different. And it also depends on whose original hypothesis it was.

A lot of times we have conversations with consumer insights professionals and they understand that research is research and that the data is telling you one thing. You can still make a different decision, but the data may be telling you something different. However, not everybody all the way up the ladders into the C-suites necessarily understand that. So we try to coach our partners and our clients in the insights world of how to have those even more difficult conversations when we have just proved the CEO wrong.

Those are much more difficult conversations. And especially in a food tasting world, a lot of people in their companies will have what they call golden palate syndrome, where they believe they know exactly what the client wants. And you see it in durable goods and any other industry, just because I spent the most of the time in food. That's the reference that we call it is the golden palette. So it's a lot of sitting down and saying, here's what the data is saying. Here's what you thought it was going to say. Let's talk about why it's coming back different.

And truly that here's what we thought based off of everything we knew. That's the say-do gap. Consumers were telling you this. People were telling you, I don't buy Oreos. Well, we went into their house and we took a picture and how many Oreos do they have? Every kind, every flavor, including the ones from Avengers. You know, people say one thing and they want you to think the best of them. Nobody wants to lie. But, you know, if they think telling a little white lie might make you think better of them, maybe I don't buy the Oreos. Maybe I'm a super health nut and I only buy natural dog treats for my dogs and my kids have never had sugar, you know, while they're in the corner eating a box of Froot Loops, you know, straight from the box, no milk.

Priscilla McKinney: You.

Rachel Buss: And what's really great is when you do a lot of this kind of work, and for us we branded under the arch of in-market, so getting that in-context type learning, we usually have a lot of qualitative type inputs, so whether it's videos, whether it's pictures, a lot of those things that can show you visually the truth because even when people write things you can say, you know, maybe they really meant this, maybe they really meant that, but no, like here's the receipts, the visual virtual receipts of the pictures and all of those kinds of things. It's pictures. Well, I would say pictures don't lie, but now in the world of AI, maybe they do. But, you know, those kinds of things are even more important these days.

Priscilla McKinney: Right, right. Well, I actually called you about this because I really enjoyed a webinar that you did about your in-market testing. And I felt like it just had that right balance of we've got to see it to believe it. And we also have to be very rigorous to make sure that this data holds up and that we're not getting just this one experience from someone or, you know, or getting some kind of an anomaly. And I thought it was really good.

I'm going to link it in the show notes for anybody listening. If you want to hear a little bit more specifically about what she just brought up about in-market testing, please do. But to kind of wrap this up, Rachel, you're very experienced with this. I kind of want to know what you think the future of research design is. Where do you see like in context, real world research methods heading in the next few years.

Rachel Buss: Yeah, so where I think research in general is going in the next few years, I think you're going to start to see a split. I think you're going to see some level of wanting to invest more in tech and how tech can help us with our research. But I also expect to see some level of an AI backlash when the synthetic data doesn't give you the fun extremes of the woman who sat in my focus groups and told me if she hides the candy bars, the calories don't count. You don't have those anecdotes. You don't have those stories with that tech. But people still are looking for faster, cheaper, not necessarily better, just faster and cheaper.

But then when you get onto that other side, because tech is going to be so ingrained, people are going to be looking for that proof point that it is a real human experience, that it truly is what they thought it was gonna be if they used any sort of tech enabled into it. So they're gonna wanna see the video proof, the audio proof, the photos that consumers take of their things. They're gonna wanna go and have that more human connection with their consumers, go on the shop-alongs, sit with the consumers.

We saw post-COVID a boom in the digital research space because, you know, six feet apart, right, so you're not going on those shop-alongs. You're not going in their houses. We had people in an early COVID era where we did a mailer and we had on video this person picking up the package that we had mailed them wearing washing gloves and opening the package and we shipped them, I think it's parchment paper or foil or something like that. Wasn't even perishable, but they were so terrified of opening the box because that's where we were all at that point.

So now that we're starting to see it shift back the other way and we're especially starting to see that this tech backlash of like, do I need my washing machine to be Wi-Fi enabled? I don't know. It can be. I've had it for five years. I don't even know how to connect it to my Wi-Fi. What does it do? I have no idea. So you have those kinds of things, but consumers are going, you're gonna wanna be with your consumer so you can see it with your real eyes to believe it. Because now as fast and as good as AI is going to be getting, almost anything is possible.

We had a recent ethnography that somebody on my team was doing and we're really good at our recruiting. We're really good at what we do. Someone still slipped through the cracks and as they were talking, our moderator correctly identified that based off of how they were saying something, the tone that they were saying, the little slight gaps, it was clear they were feeding the questions that they were getting into some sort of ChatGPT or Gemini and then parroting out the responses. Like the responses that they were giving were just too perfect, too on the nose, missing that color. And you know, where we're like, how did this person even slip through our recruiting processes? So if we had been there face to face, like you can't do that. You see it with people interviewing, you see it all across the board. So I really expect to see this huge push for in-person, face to face, side by side, even if it's just digital, something of that nature to really validate the human experience and making sure that's actually being built into your data.

Priscilla McKinney: I love it. And you know, just to validate my human experience, I will connect my washer to the wifi if it will wash all the clothes and fold them. So if that's on the docket, like, yes. I don't even know how. I don't know. I don't know. Well, you can tell from talking with Rachel, how much of an expert she is on here. Please reach out to her on LinkedIn. It's R-A-C-H-E-L. Buss, B-U-S-S, so connect with her there, check out our show notes, I'll add a couple of other things, I'll hit Rachel up for a couple of other goodies and takeaways so if this is your area of interest don't leave the podcast without taking a look at the show notes. Rachel, thank you so much for joining me.

Rachel Buss: Yeah, it's always a great time to chat.

Priscilla McKinney: From all of the peeps here at Little Bird Marketing, have a great day and happy marketing.