Mastering AI as a Data Quality Tool

Priscilla McKinney: Hello and welcome to Ponderings from the Perch, the Little Bird Marketing Company podcast. I'm Priscilla McKinney, CEO and Mama Bird, over here with you as always. It is my job, and it's also my pleasure to invite friends of mine in the industry, people who are leading the way with how we are addressing some of the challenges in this industry. And for those of you who are running and running, this is your show to get a chance to catch up a little bit with what is going on in market research.

And marketing and say, wait a minute, let me hear it directly from the pro and let me also take a moment to ask myself some good questions. And then at the end we want to leave you with some options and really have you hear from someone who's in the front lines. What could you do today to make something a little bit better about what you do? So I have with me today Steve Male from the Logic Group. Welcome to the show.

Steve Male: Well thanks for having me, Priscilla. It's great to be here.

Priscilla McKinney: Well, I get to see you around at conferences all over the world. So a little bit unfair. So we get to have a lot of interesting conversations. And I gotta tell you, of course, the moment you know I walk up and talk with Steve Male, the next thing we're gonna say is something about data quality. But if anything, from when you wrote your book about AI, the AI imperative, and like what does this mean for market research?

Whether you're programming something into some of the powerful tools at the Logic Group, or whether you're actually in a meeting with clients looking at the data and trying to get to the bottom of what's going on so that these global, massive or regional and tiny studies can be the absolute best they are. This is where you and I end up talking. And I do prefer it when we have a glass of wine in our hands.

Steve Male: It always makes the conversation better.

Priscilla McKinney: I love it. I love it. Okay. So before we get started, if people don't know you, Steve, give us just a couple sentences about your day to day at the Logic Group so they understand kind of what you do and how the Logic Group helps with you know, actual operational, you know, back of house in this industry.

Steve Male: Sure, yeah. So at Logit we tend to work with a multitude of clients and we do everything from when we receive a questionnaire, we do fielding programming, data quality, anything to the point where we hand back the data for analysis for our clients, but we handle all the field services in between.

The way I always describe my role at Logit, is I get to work with the cool new technology and figure out what to do with it before anyone else does. So it's a little bit of fun for me and my team, but generally we're trying to find ways that we can use AI or technology efficiently to solve real-world problems that our clients are facing.

Priscilla McKinney: Right, right. I know this, you know, instead of like theoretical kind of stuff, every day your team is like, yeah, but how are we gonna create a sprint to solve this issue? This is new, either newly emerging or incredibly persistent problem in this industry. And how can we continue to hack at it and really make a difference for our clients and be proud of the data that we're handing back out? I think that's so key because people will keep coming back to Logit.

And for anybody listening, of course, your clients are gonna keep coming back if they trust those results and the decisions that they're making in business are working because they have been really informed by consumer voice. And so, Steve, let's talk a little bit for a minute about the theoretical, kind of the larger problems, level set a little bit about the challenges you know brands are facing. It's brands, but then for you that trickles over to the market research firm that then basically hires you to handle the back end.

So you understand what the problems brands are facing, but you also understand the problem that market research firms are facing. So let's talk about those theoretical challenges first and then we'll get into some specifics.

Steve Male: Sure, so generally I get two questions all the time from clients or when we're at conferences and shows. The first question is what are you doing as a company with AI? How are you utilizing it for the work that you're doing for your clients? And two, how are you handling data quality? Both are really big issues in our space.

And I think we're at a point now where everyone is aware that data quality is a growing concern and issue, and people are cognizant that we need to work together as an industry to make effective change to reduce the impact of that. And I think we're also at a point now where it's no longer a question of is AI going to be adopted and utilized? Everyone's doing it, but now it's a question of how do we do it efficiently and what how does it make sense within our current workflow process. So for us, those are kind of the hypothetical theological things that we use as starting points for our day-to-day work at Logit when we're creating solutions to problems that we're seeing on the client side.

Priscilla McKinney: Okay, I love that. And I have to say, one thing I mentioned was there are newly emerging problems, but there are persistent problems that don't seem to be going away. And then we added this AI layer to it. And those challenges have not gone away, but yet it's created this expectation that people can not only solve these challenges with AI, but they can be solved faster. So kind of give me that exacerbated challenge. Like what does that sound like? What does that feel like? You know, what are people saying to you about that?

Steve Male: Well I think AI is the greatest piece of technology that civilization has created since the internet, right? It's really leveled the playing field and nowadays, you know, if you're not a coder, if you don't program, you can go in and vibe code and build yourself an application in an hour or so. It's really phenomenal what it can do.

But one of the things that AI doesn't really do a good job at is making up for years and years of experience or methodology knowledge, it doesn't really have that baked in. So I think one of the challenges that we see, and I hear it when I talk to clients, and I hear it when I go to conferences and shows, is that people think that they can go and acquire an AI product and it's gonna be a magic bullet. They're just gonna be able to drop it into their ecosystem, turn it on, and then it's gonna solve whatever problem they have from curating moderating interviews and diving into insights, solving data quality, you name it.

But from what we've found through a multitude of tests and the work that we've done with clients is that's not the case. AI is simply a tool in your toolkit and if you're starting with an inefficient workflow or process, it's going to just amplify that. It needs to be dropped in to a process that's been thought out and it really needs to augment and complement that process.

Priscilla McKinney: Okay. I love it when someone starts talking about processes. I'm such a process oriented person. And I gotta say, of course you guys are because you're taking someone else's work on, taking it and saying, okay, well, they don't have the global reach or the footprint or maybe the personnel to handle this globally or regionally, and so you're taking that on. Well, of course you need to have processes. You have to be very dialed into standard operating processes so that you know that you're getting the same thing each time and obviously delivering on time too.

But let's talk about the bad news of AI as it relates to data quality and what we're talking about very specifically in those challenges. I want to do that because I want to end on the good news. Everybody hang in there. We're gonna get there. But the bad news, what is happening with AI real world that is making these challenges worse?

Steve Male: Well, I mentioned that AI is a great tool to have in your toolkit. I guess the bad news is that poor quality respondents, bad actors also have AI in their toolkit. And as fast as we're working to implement processes and tools to detect that poor quality, they're working on the opposite end to use it to enhance the type of responses that they're generating in our survey environment.

Priscilla McKinney: Well, okay, tell me a little bit more about that. What are you really seeing? What are bad actors? What is reality? Because you and I know we talk about this a lot. People cry a lot about bots. Bots are doing this, bots are doing that. Okay. Is that really the problem? Tell me what's really happening.

Steve Male: Well I think bots kind of get a bad rep. I mean, bots are bad, and bots are still part of the challenges that we see. But when you look under the hood and you look to see, you know, we tend to group poor quality respondents or bad quality into one bucket. They're really different forms of bad quality that we're seeing. And bots account for probably less than 10% of that across the whole survey ecosystem.

One of the biggest problems or challenges that we face as an industry are what we call survey farms. So essentially these are an offshore conglomerate of individuals who've gotten together. One person has gotten the questionnaire, the screener, they found their way through, and they use all these tools to come across as respondents that would be good qualified candidates for the survey itself. So they're using AI to perfect and enhance the open-ended responses that you're seeing. They'll use it to do knowledge and deep dive into brands or find out correlations between questions to make the answers that they're providing make sense.

So a lot of the traps that we used to bake in over a period of time are no longer relevant because it's no longer finding bad quality respondents. Now it's almost finding people who the responses are just too perfect. We actually had one client and we were joking, you know, years ago if anyone said not sure, I don't know, they would probably be removed or screened. Nowadays you almost want to see those types of responses because you know they're the real respondents or human beings taking part.

Priscilla McKinney: Right, instead of hearing furthermore, and in conclusion, I sincerely think that this is what we should do. Right. So either that or the poor dash, right? Like how do we spot AI? And I'm so sad to see the dash die, but that's for another topic. But what you're saying there is that it is almost, you know, because it is such a powerful tool, people are wielding it against market research firms in order to cheat the system.

And so where people were looking, or the kind of response they were looking for, is maybe kind of not the response they should be looking for. It's not the marker anymore of fraud. Now it looks different. And so people have had to adapt and adopt new processes. What has the Logit Group done to adapt to that?

Steve Male: Well, the way I look at it is a lot of what we see in our data sets from the open ends to the interconnected logic behind responses on the close in the questions, they're all symptoms of what is really happening behind the scenes, right? So content and looking at content that's delivered in our survey data is still very important, still part of what we do.

But in addition to that, we now look a lot more at metadata. So what we're collecting on responses from you know an IP browser operating system, is that changing over time? So we have fingerprints on that. And more importantly, how they're interacting with the survey environment itself. So we can see people who have multiple tabs open if they're hot tabbing back and forth, probably looking up how to translate or using an AI agent. And we can also see how they're interacting with various questions, the time they spend on each question, whether there's sufficient mouse movement or keyboard strokes, and whether they're copying and pasting in answers.

So basically what we're seeing behind the scenes from a behavior standpoint is now just as important, if not more important, than the content we're seeing from these individuals within the survey.

Priscilla McKinney: Okay. So what I'm hearing here, and I think where you and I end up having a conversation often when we're face to face is that it's not one thing. There's no silver bullet. The fraud issue is not just fraud, there's data quality issues, even if you have the right kind of person and they are legitimately someone you want to hear from, it's also about getting a quality response from them.

Steve Male: Yeah, and from what we've found, we've done a lot of research on this, no one data quality tool is a silver bullet. It's a combination. And usually once you overlay a number of different detections, that's when you start seeing trends or patterns. And we find on average, a poor quality respondent or someone that's going to be removed from a data set is tripping upwards of four to six of these on any given study. So layering that in is really important to detect these individuals.

Priscilla McKinney: Okay, I love that. Let's stick to the bad news for a minute, even though it's very, you know, not like me. I'm a very positive person. But let's stick with the bad news for a minute because we talked about how okay, these are the challenges we're facing. And now AI is bringing in powerful tools, powerful tools against us, but powerful tools that we can wield as well. And so teams are now implementing AI.

But how is that going in reality? Because getting the tools is only one part of the equation, putting them in place in the right configuration, as we all know, no matter what it is, is so much a part of the real winning structure. So tell me what you're seeing people trying to do, trying to implement, like what do you see some pitfalls or what's going on when they start implementing AI?

Steve Male: Well I think there are, and don't be wrong, there are some companies that are doing it quite well, but I think there's a lot of other companies that need improvement in terms of the tools that they're rolling out. So for us at Logit, we've been in the research space for almost 30 years, and I think we're at an interesting crossroads where we have 30 years of research acumen and experience that we can overlay with new technology.

So the way my team and I approach any type of task is as opposed to just hammering a square peg into a round hole, is this tool going to work within our current workflow and process? And if it's a no, we go back to the drawing table, rethink it. If yes, only then can we proceed. So one example is we all work with sample providers, partners that will help source respondents. There are tools out there that will let respondents run through the entire survey once they're completed, then they'll look to see okay did the results and the data that we received from them meet a certain quality threshold. If no, then send them back to the panel source.

At that point in time, they've already completed a 20-25-minute survey, right? Imagine trying to do that on repeat with your survey partner over time. They're probably gonna stop wanting to work with you. Whereas on our side, we do that in real time in front of the survey to make sure if they are failing those quality thresholds, they're passed back in real time and not impacting the overall length of interview of the survey. So there's a lot of things like that out there that just makes a lot of sense from a logical standpoint. If it's having negative impact to the survey or the partners that you're working with, it's not a sustainable process.

Priscilla McKinney: What you're saying there is that people have to be very awake and alive to their own workflows and their own processes. And they also have to be paying attention to the experience for the respondents. Part of us being able to maintain a good ecosystem of getting quality data from consumers is making it a good experience for them. And if they get through a long process and then get disqualified at a weird part, it doesn't help us in the long run.

And so I guess I'm wondering what you're hearing from people, you know, are they aware of where they should implement AI, where it would be helpful in their workflow.

Steve Male: And I think that's one of the things now, like I was mentioning, everyone knows that they need to implement AI in some way, shape, or form. But I think only now are we starting to have that wherewithal to say, wait a second, is this making a lot of sense, right? And I know we as an industry get up in arms and say the bots are coming, there's poor quality respondents everywhere. But there's still a small part of our ecosystem, right? And I think we tend to get a little bit adversarial sometimes and say it's us against respondents.

But to your point, we need those respondents. They're the backbone of the research that we do, and without them, there's no data for us and our clients. So part of that is understanding that the AI has to complement that user flow and be injected at certain key points where it maximizes impact to the quality of the data, but minimizes impact to the respondent.

Priscilla McKinney: Yeah. So putting it in at the wrong time can be frustrating and really cause a breakdown in the project. And that could be for the respondent, but it also could be for your team. I mean, I think about the frustration if they have to re-field something or they get something back very late in the game. And we all know about time compression on getting these surveys fielded, especially the way you all work globally, the time compression's a real deal.

And you know, that's one of those things that AI has created these crazy expectations of just how fast everything can be done. And yet if you want it to be really quality, even with using AI, you can cause a pretty major breakdown in the system, maybe a system that was working okay the way it was before you implemented AI.

Steve Male: Well, a hundred percent. I mean we work with a lot of partners and clients and again I think we've kind of come to this conclusion or thinking that AI is a magic bullet. We can drop it in and just go off and grab a coffee and come back and it'll be done. It's not the case. We find, when we're working with AI as part of our workflow, we'll let AI do eighty, ninety percent of the heavy lifting and then we'll have human interaction for the remaining ten, twenty percent.

So it's not really a conversation of, and I hear this a lot, is AI gonna take our jobs? Probably not, but it's going to augment a lot of what we do and really complement that process so that we're spending time on things that matter and letting it do a lot of that repetitive templated tasks that were manually spent before.

Priscilla McKinney: Right. Okay. In your role as the executive VP of innovation and strategic partnerships, and I highlight this and underscore this right now because this is where you sit in the industry. I do get the sense from you that you feel a duty of care to our industry and you speak out pretty boldly. Obviously, you wrote a book about AI and what it can reward us with and what those risks are in this industry.

But from your perspective, how do we move from that bad news we've been talking about, which is the potential breakdown of systems and maybe a loss with respondents, maybe even of trust with our clients, even our good clients that have trusted us for a long time? How do you advise people to move from that and into the good news of AI?

Steve Male: Well I think like anything, when bad news comes up, we tend to focus and hone in on it and I see it less of a challenge and more of an opportunity, right? So now we have this great tool, we have AI, we're starting to do really great things with it from AI moderated interviews, we're looking at it across the survey ecosystem to determine quality responses.

To me now the question is how do we double down on further improving the workflow, right? I think it goes back to what we're talking about, the respondent or user experience, right? How do we stop thinking about hey, how can we make sure this person is a legitimate respondent and take that same technology and say, how do we give this person the ideal experience and journey through our respondent ecosystem to give us the best data we possibly can. I think there are some companies that are starting to do that, but I think as an industry as a whole, we gotta stop dwelling on the negative and starting to focus on how do we use this technology for good.

Priscilla McKinney: I love the technology for good concept. And if you think about some of the things you've implemented more recently at the Logit Group, I know it's like a constant tinkering, but can you share a couple of examples of some ways that you have been able to come in and bring AI? I'm not asking for trade secrets here, but I really, because you are so involved in the actual innovation, in the actual programming, in the actual engineering of what's going on. I think you are such an authority to be able to speak precisely to this is what we are doing, and this is why it is making a difference.

Steve Male: Yeah, so we looked across our entire workflow to see where we can really inject AI to help create efficiencies in the process. So over the past year we've done things from taking a PDF or a Word document from our clients and then as opposed to handing it to a survey programmer to inject, we'll now get AI to parse it all out, create XML, we can just inject it right into our survey authoring instrument. So we can go basically from a Word document to a programmed survey in ten minutes or less and then we can spend a lot of the time testing and fine-tuning, as opposed to doing a lot of the heavy lifting.

We now use it for all the coding work that we do so we can create themes and look at open-end responses on the fly and do a lot of that heavy lifting. And more recently, on the phone side, and believe it or not, we still do a lot of phone work. I know it's not necessarily as prominent in the space as it once was, but we're now looking at ways to augment a lot of what we do with human interviewers and we're turning to AI for that. So there's a lot of really great testing that's in place right now and we're really excited in terms of what we can do with that in the near future.

Priscilla McKinney: Okay. I love that. If you could change something in the industry and you could say, look, this is why it's worth working with the Logit Group. There may be, you know, like you mentioned, I love that being fair. There's some people out there that are doing it well also. It's not a matter of nobody's doing it right. But the Logit Group, what would you like to say to people about that? Like if you are going to have a quality conversation with your operations, with your fieldwork company, what should you be asking about AI? How can they take something practical and positive today and say, well, let's have a better conversation about what's going on? What would you advise them to do?

Steve Male: Well I think to your point it's now about asking probing questions. I think in the past, we just showed up at shows and we got emails from providers and they said yeah, we can provide you this service and we said great, here's our work, go ahead and do it. But I think now it's more a question of, I always suggest to our clients asking on three things.

So one, how are they using the AI? What's the workflow and process on their side, and truly understanding how it interacts or injects within what you're currently doing. Does it augment or does it contradict? That's important to know. Number two, if they are utilizing AI, how are they handling the data that they're collecting from your side from a privacy and confidentiality perspective? Because there's a lot of things that we have in place on our side at the Logit Group in terms of how we build our tools, but not all companies take that into account.

So really having an in-depth conversation on that and truly understanding if I'm sending them data, is it being used for training? Is it being sent into the larger cloud or is it being protected? And number three, I think, and this might sound crazy to say, but is the tool actually using AI? There's so many tools out there that brand themselves as AI tools and when you open the hood, it's not. It's just kind of an innovation process that has some efficiency wrapped around it, but not necessarily AI. So I think the more you have conversations with providers and I'm fully advocating that we have more transparency on this in the industry as a whole. But the more we interact with providers and start to understand truly how they're using AI and how it relates to what we're doing on our side, the better. And I think before that was nice to have, now it's a must-need.

Priscilla McKinney: Yeah, but I'm just gonna push back on you a little bit. We're all really pressed for time. And you know, sometimes people don't even know they have bad data quality. So can you give people a little bit of like what should you be looking for? Like, should you be having a conversation? Like if your provider hasn't talked with you about X or if you notice this, you know, there are some of those symptoms. Obviously, you're saying hey, don't just treat the symptoms, like look at really what the root causes are. What are some of those symptoms or red flags so that people would go, okay, wait a minute, I haven't had the right conversation.

Steve Male: Well, I think if we're talking from a data quality perspective or sample perspective, some of the red flags that you can see without having any tools behind it is just in terms of when does your sample come in. Right? If you're running a sample in North American Eastern hours, if we're seeing sample come in overnight, that's kind of a red flag.

Look at the survey responses themselves, so like over elongated open ends is usually a telltale sign that something is going on there. And then one of the things that we've done, and it's again it's silly to say, but if you put an open-ended question at the end of your survey and you ask people about the survey experience itself, it's funny you'll find people often drop their guard and start giving you exact answers of where they're coming from and what they're doing. It's really telltale so we use that on our side as well.

But I think it's important to go back to your providers and have these tough conversations and ask, you know, if you're seeing data that doesn't make sense, have that conversation and demand transparency and expect quality responses in terms of describing what they're seeing and how this data is being compiled.

Priscilla McKinney: Right, right. You know what, you've gotta be having these conversations, even though we're all pressed for time because this data quality thing is just all the difference of making a multimillion dollar mistake or not. And I know that you all work in some smaller projects, but some of the projects you're working on are so massive. They're global scale, companies are about to commit major budget to it.

And getting some of these things wrong, I mean, there's no going back, you know? And it is a little bit frightening to me to think that some of these companies aren't demanding this kind of transparency from the get go.

Steve Male: Yeah, I mean we run projects in eighty plus countries. There's one large project that we do where yeah, there's a lot riding on the data that we collect, so it's gotta be right. And I think for us, like I mentioned, we have thirty plus years of research experience that we tie into the technology that we're building. So we understand the impact of this on the data and what that data means to our clients.

So when you're having conversations with providers out there, ask them about their background in the space. You know, how long have they been in research? Did they come from a research background? Even those types of questions kind of give you a sense of how they're interacting or working with the data that they're collecting.

Priscilla McKinney: Yeah, I love that. But I just want to give you guys at the Logit Group a shout-out, not only because you guys are super nice, but you're always approachable about these issues. You guys are just so nice. You don't throw anybody under the bus. I love how you said earlier, hey, some people are doing it right and we are in a good peer group. But I love how you're willing to also just be helpful in a space where we all are meeting challenges and being willing to say, well, this is what we're doing. Let's pop the hood. We'll show you so that we all can do better.

Because I do think that is what is going to keep the industry trusted. And I think that's one thing that people don't think about a lot. When we hold some of our cards a little too close to the vest, we run the risk of the outside industries looking in and then not trusting us over time. And that is a scary idea. So I feel like the Logit Group really plays into that. This is why I've asked you to come on my podcast because I believe in always being helpful to people. And I just want to give you a shout out for sometimes when I hear you guys speaking at industry events, it's just not this crazy sales pitch. You're about helping people make a better decision about what we're doing in the industry so that we can all go forward. Just a huge thank you to you guys.

Steve Male: I appreciate that and I always love a good challenge or a good problem. So whenever we're speaking at events or even on the trade show floors, we always encourage people, you know, even if it's something outside the scope or wheelhouse of what Logit provides, ask us. I'm happy to be as transparent as I possibly can and provide insights to a potential solution.

Priscilla McKinney: Yeah. Well, I think it's really valuable from your point of view, from the footprint that you guys work in, which is incredibly large. It must be very complex, pretty difficult to manage around the globe. But I do think that you have that singular focus and that's super helpful. Steve, thank you so much for joining me on the show.

Steve Male: Well thanks again, I appreciate being here and lovely as always talking with you.

Priscilla McKinney: Well, from all of the peeps here at Little Bird Marketing, have a great day and happy marketing.