< See all episodes

Data driven decision making with Uber Eat’s Chip Koziara

Episode #

18

In this episode we’re joined by Chip Koziara. Chip is the Manager of Strategy and Planning at Uber Eats and leads the Deal Desk team, which is responsible for Uber Eats’ restaurant partnership strategy and pricing in the United States and Canada. We discussed:

  • Being data driven with your team
  • When do you know you have enough data and are ready to make a decision
  • How to use search data to augment your inventory selections
  • How to make a business case once you have data to support a decision
  • How to get started with data driven decision making

Resources

Getting in touch with Chip

Stuart Balcombe
Hello, and welcome to the Customer Conversations Podcast. Today I'm excited to be joined by Chip Koziara. Chip leads the Deal Desk team at Uber, and is responsible for Uber Eats restaurant partnership strategy and pricing in the US and Canada. Chip, thanks so much for joining me.

Chip Koziara
Thanks for having me on, Stuart. Super excited to catch up with you today.

Stuart Balcombe
Of course, yeah. It's going to be a great conversation. I think we're going to get into lots of interesting stuff around customer data and how you use that to help inform the strategy and the things that you're doing at Uber. So maybe we could start off for those who don't know, what does it mean to lead the Deal Desk team at Uber and what does that actually entail?

Chip Koziara
Yeah, it's a great question. So when I first applied to that team at Uber, I also didn't have a lot of familiarity with what that term meant. And for quick background, I joined Uber in a marketing operations role focused on building different tools to help our marketing team's scale, and I was focused on the consumer side. And for those of you who don't follow Eats super closely, we have a three-sided marketplace. We're focused on connecting eaters, the restaurants, and having the food from these restaurants delivered from couriers. So we have these three different types of customers that different parts of our businesses and different teams touch. And so I was focused on the consumer side, or what we call eaters, initially. And then about a year into my role, I had the opportunity to apply to, and then ultimately join the Deal Desk team. And I thought the name sounded kind of funny and I was like, "Hey, what's this?"

Chip Koziara
But it's really all about focusing on merchant pricing. So making sure that we're making deals and setting deal terms with specific partners in a way that makes sense for both of our businesses. And so any time we entered … It started off more with partnerships with large enterprise restaurants, but the team's scope now includes all restaurant partners in the US and we work across a bunch of different teams to make that all happen.

Stuart Balcombe
Cool. Yeah, I'm sure in current times, recording this in May 2020, we'll see how that ages. But particularly relevant with companies trying to go to a take away or order-for-pick-up models. I'm sure that's impacting the things that you're doing. But I think there's a really nice segue here that I think what will be the meat of this conversation which is, how do you use data to make those decisions. Your goal is to make decisions that will work in the interest of both your business and your customers'. Where does customer data, and I know you're Uber, so you're working at Uber where you have a ton of data. So how do you go from, you have all this data to, it's actually helpful in informing our strategy?

Chip Koziara
Yeah, it's a great question. So I'll start by defining what types of strategic questions we think about, and then backing to how you see it to solve those things, and I think a lot of this applies to platform businesses more broadly.
So what I'm really focused on is what I call selection strategy. And so I think of selection, which is a common word but applying it in this way I think of it as, the availability of distinct goods, services, or service providers that a business offers to its customers. So like with Netflix, you'd think of their selection as the different movies, TV shows, and documentaries they make available to their customers in a platform business. At Uber Eats, it's the restaurants that we provide to our customers. And then someone like an Amazon it'd be the different goods and things that they make available to their customers.

So really focused on solving that selection strategy problem. And so kind of working backward from there, you have to think through, "What do our customers want?" And that's really where a lot of this comes into play in terms of thinking about how do we use data to inform a lot of those questions.
Do you want me to jump into a couple of different ways I think about that or … ?

Stuart Balcombe
Yeah, I think that would be great if you have an … sort of follow the data, I guess is maybe a good way to think about it. Sort of all the way from how do you collect it, to how does it actually impact that strategy?

Chip Koziara
Definitely, definitely. Depending on what type of selection problems you're trying to solve, a lot of it might be if you're in an industry where your selection is pretty straightforward. So if you are a florist and you need to provide flowers, pretty clear. There's only so many different types of flowers in the world. I don't know how many there are but there's some way to quantify that. And so you'd have that available to you and you could then figure out, based on different demand patterns, maybe you buy data sets on what flowers are the most popular, back into what selection could look like for your flower shop.

Another way to approach it … So that's going through the external data source or sources. Another way to approach that, continuing the florist metaphor, if the florist has a website and there's a search bar on the website and someone types in, "I want two dozen roses." The florist would then have that search string of what someone actually wanted. And so you're getting a revealed preference almost where someone's typing in a freeform text box actually, "This is what I want." And so you're getting a data point that's not really a survey because there wasn't a menu of options available. It's someone who is presumably at or close to the point of purchase. So they're giving you data that's pretty interesting. And then you'd have to do a lot of things on the backend to make that data usable and more structured. But that's another super interesting way to go about it.

And then you can look at what your competitors are doing. So you could look at the flower shop down the street or across town and see what they're making available to you. So I think those are a few different ways that that problem can approached.

I think one of the things that doesn't fully solve is if you're in a new space where people might not know what options are available. And so if you're on Netflix and you're looking for … Like I don't think anyone knew Stranger Things. I don't think anyone was like, "I'm looking for a retro, 80s, horror TV show that's based on a bunch of kids' lives." That wasn't something people were searching for presumably. Maybe they were. So I think it's not a perfect solve for this is everything you'd ever need but I think it's a really, really good start.

Stuart Balcombe
Right. Yeah. That totally makes sense. And I guess there are some other ways to get once you have people, once you have customers who you're able to have a conversation with are able to either your reviews or, and you mentioned competitor data as well, reviews of competitors … Future requests is maybe not the right word in the florist example but, "I got this but I would have liked X. So can I also get Y?" Because you start to be able to identify those additional data sources.

So, you mentioned that on the back end you have to go through a cleaning process or at least there's some work to be done to go from there's all this data that exists that I have access to, to I can actually use that to inform my business. Can you speak to that a little bit? I know we talked before, sort of nerding out, about some of the actual ways that we do that with text analysis and the bigger question I guess, is quantitative versus qualitative and how the mix of those things comes together. But maybe just speak to, and maybe continue to use the florist example, I like this example. It's such a non-tech example but yeah, speak to that a little bit.

Chip Koziara
So I think using the florist again, say the florist has access to their database and they're super excited to start writing some queries or looking at the tables, they can pull out all those different searches and look at that. Then again, you're going to have two dozen roses, hydrangeas … I don't actually know too much about flowers so I'm going to start to run out of … daisies for another flower example. But you'd have all those in an unstructured format. And I think, depending on how big or small your data is, if it's your florist maybe you're only getting a dozen searches a week where you can manually go through that and throw it into a Google Sheet and make some sort of ranking where maybe you're getting a third of your searches are roses and two of them are daffodils and then the longer tail of random, assorted flowers.

So in that case I think it would be take a lighter approach. Pull the data out, throw it into something like Sheets or Excel and crank on it that way. I think once things start to get later … So I think earlier in a company's life cycle, probably that more manual approach is probably best. You looking at things, every entry and every record. As a company starts to scale, you're going to not totally be able to do that. I think it's always helpful to look at a slice of data on a certain day or something like that and just see what's all in here. But I think that's where having those industry data sets … So using the flower repository, these are all the different flowers that we could have, and use that as your source of truth for possible skews or possible available selection, and then mapping the search strings to that.

And so a few different ways to do that. You can write a bunch of SQL case whens to make everything fit in that. And if you only have 120 different types of flowers or skews, that's actually probably a pretty good approach is to just keep that business logic in SQL and do it that way. If the search base, I guess, of all those different skews is much broader or changes frequently, that's when I think starting to do some more text analysis and pulling in some Python libraries or R libraries to start to crank a little bit more, comes in handy.

Stuart Balcombe
Yeah. I think it's interesting to think about volume of data. And there's always, well, usually at least, ways to get more data, right? We've talked about the search box on your individual small business website maybe isn't getting a whole lot of traffic period. Whereas Google is getting a lot of traffic. And you can do a local search on Google or whatever you want to do. And I think it's interesting to identify the specific things that you're looking for. You're not really looking for the exact terms that the customer uses to … which helps on a couple of fronts. A, it helps you determine this is the thing that they're looking for. I need to make sure that I have that in my selection, assuming that there's enough people who are searching for that, but it also gives you some context around that.

I guess less so in search queries, because they tend to be pretty specific. But in things like reviews, or you may get those in support tickets or in chat transcripts is, "Why am I looking for this thing?" Or, "Where does this fit? I'm looking for this thing." And I guess you get this from search too. But, "I'm looking for this particular thing at 9:00 AM on a weekday only." That's where your volume comes from.

So I'm curious, how do you think about mapping? Once you get really good into the data, you can totally get lost in the data, right? There's so many different things that you can pull out. Where do you say enough is enough? I have the right amount of data. I have enough signal to move forward and go to the next step and actually start to use that data.

Chip Koziara
Yes. So definitely will answer that question, but I just thought of something as you were talking. So let's say in our example again, someone's typing in "Valentine's Day" instead of typing "hydrangeas." That's interesting because their job is, "I want something nice for Valentine's Day." Not, "I want this very specific type of flower." And so if you just had all the skews of possible flowers, you're going to miss that.

And so I think that's why looking at slices of data and of the raw data and understanding how … if it's together and trying to back into what people are thinking about and what problems they're trying to solve, what their jobs are, then I think you can have better mapping. And just knowing that someone's looking for Valentine's Day, isn't going to give you … that's not a skew that you can just throw up on the website. But I think that's helpful for a question that you could use in a customer interview where it's like, "Hey, I noticed that you searched for Valentine's Day. Could you please tell me more?" And so I think that's where that can come into play too.

But to your question on when is enough, enough. I think it depends on a lot of different things. I'm of the camp 80-20 it were let's figure out what's going to give us the strongest signal as quickly as possible and then start to move on. We can optimize later. And so for me … I'm not a data scientist. So for me it's just like, let's do the quick-and-dirty way to get to a great answer and then optimize later. So at that point, it's hard to know. It's more of an art than a science. But really once you feel that you have a pretty accurate understanding of what your customers want and can pull that together and synthesize that in a digestible format.

And so what I think that would look like is a pivot table or something like that, where you have all the different products or merchants, or types of flowers or packages, and can see how many people search for how many different ones in a way that's meaningful. So if you can see that over the last year 832 people searched for roses and 645 people searched for peonies and 4 people searched for daffodils, maybe daffodils isn't something that needs to be a big focus. And you'd move on to the next phase, which is what I think of as actually going out there and acquiring the selection.

And so depending on the org you're in, the company you're in, if you're a solo founder you're going to be doing all that work too. If you're in a bigger org, maybe it's something that sales or BD would take on. And so you essentially give them the customer priorities and say, "Hey, these are the things that we need to bring on." So like for Netflix, maybe they saw everyone's searching for Seinfeld and decided that that was … I bet Netflix definitely looked at those logs and saw it's worth bidding for Seinfeld. And so they were able to hand that to the sales team and say, or the finance team and say, "This is justified because we had millions of people searching for a show that we don't offer." And so I think empowering other teams with data is that next step. Or if you're a solo founder, empowering yourself with data to take that next step.

Stuart Balcombe
Right. Yeah, it's interesting to think … The way that I think about this is that it's like an efficiency optimizer, right? The more data that you have and the … and this is why it's a scale. There's different levels. I really like the sort of 80-20 stance that you can go all the way if you want. You can … well, there sort of is no all the way until you really get it out in the world and test it. But you want to go to the point where you feel confident that this is a better bet than something else that we could be spending the same time on.

So, I'm curious, at Uber, and maybe you can give an example here, how do you present that case? It's sort of, "I've done the research. I've done the digging through the data. I now have to give that to somebody else and convince them that this is the right path or that this is valid enough to take on." How does that process look like? What does that inner team communication or even within the team look like?

Chip Koziara
Totally. Yeah, it's a great question. So really just synthesizing the data and making a business case and especially compared to the alternatives. So essentially like an opportunity cost. So, let's say that again with the flower example, let's say that I am recommending to our master florists what they need to stock, I'd say, "The roses is being searched for much more frequently," and you'd give the number, "than the daffodils, therefore we need to either get more rose merchants, or we need to figure out a better way to stock it," and showing what that data looks like.

And I think it's not just the number of searches. I think you can take it a step further and think through, of searchers who are looking for roses, how many of them go on to convert on other items? And if they drop off, that's interesting. So maybe the peony example I gave, I think I said peonies earlier where the people are only searching for or whatever it was, or searching for four of them. Maybe those users all drop off. And so, yes, it's only four, but if conversion is effectively 0% for those types of searchers, that's also interesting.

So I think painting that full picture of their customers that we are losing, who want the very specific thing that we don't offer. And so, yes, there's a big business opportunity to go after options A, B, or C. But they're not as big when you dig into the numbers because those customers are going to buy something else if we don't. Whereas customers who want option D, they're gone. They're going to go to a competitor, they're going to go do something else if we don't get them.
So I think painting those alternatives of what the business impact is and what the effect on your customers or potential customers would be, is super, super important. And then I think having, in terms of how it's communicated, I think it's really just empowering the other teams to understand what customers want and to be able to prioritize their book of business accordingly. I think we can talk a lot about sales incentives and things like that. There's some interesting things you can do there, but I think that's a bit of a rabbit hole.

Stuart Balcombe
Yeah. I feel like there's a lot of topics there which we could go on for hours about. I think one thing that comes to mind as you're talking about the follow-on impact of a decision and why it's so important to go beyond just scratching the surface in a lot of cases.

We had Kristen LaFrance on in the second episode of this podcast and that conversation was all about retention-based acquisition, which is very related here. The type of customer that you acquire upfront has a … do you target and then subsequently acquire, initially has a knock-on effect on … Is that a customer who continues to purchase? Do they stay subscribed if you're a recurring subscription model? Do they go on to refer people? Or are they just there for a one-time, one-off, in-and-leave type experience? Which might look good initially in the data, but potentially if their motivations aren't what is best for your business or don't match what your business best provides, it's not necessarily a long-term win.

Chip Koziara
Totally agree.

Stuart Balcombe
Cool. Yeah, so one thing that I'm really curious about and I think is particularly relevant to ask you, because Uber Eats has so much data compared to a lot of businesses both in SaaS and in other consumer businesses. But what's the right mix of quantitative versus qualitative data? Because I know a lot of people when you talk about user interviews or, let's do 5 or 10 or even 20 user interviews in a given period. Well, that can't possibly be enough because we have hundreds of thousands or millions of users, so we have to go to quantitative data. Where do you think there's … And I'm certainly not advocating that everybody should be doing only user interviews, but where do you find is the right balance between looking at what people are saying versus just the numbers of when they search or how frequently they select X?

Chip Koziara
Totally. That's a great question. So I think this one's tricky. The way that I think about this, which is based on my experience in early stage companies pre-Uber, and now being at a big public company, I think the user interviews matter more, and I can unpack that a little bit more pre-product market fit.

So before you have product market fit, having those qualitative signals and understanding how a user perceives your product, I think are super, super valuable data points for informing product roadmap, business model, a lot of things that just require more feedback and hands-on experience. And I think the data that you're going to have prior to product-to-market fit is probably going to be much, much noisier. And so you're going to probably be looking for signal and noise, as opposed to really digging into what a specific customer's needs are and unpacking how your product or service is solving those things.

So I think once you're product market fit, I think talking with users is still very important, but I think it's more of a Basie-and-bandit-type thing where you're doing a sampling randomly to make sure that you're still on the right track. Where the data is going to inform you toward optimization, I think you mentioned before efficiency. It's going to help you become more efficient. But it's not necessarily going to point you … It's not a good compass. It's a good measuring device, but it's not a good compass in terms of showing you where you exactly need to go from a strategic direction perspective. I think it's more of an optimization tool.

Stuart Balcombe
Gotcha. Interesting. Yeah. And I think, as with everything, it always depends, and the stage of company and I guess also how much data you already have about your customer. How much you've been optimizing to date all plays in. I will say that's certainly one thing that is … once you're at some kind of scale where you have data readily available, whether you just have a lot of traffic to your website, so it's easy to survey a sample of those people, or you have customers who are leaving reviews, or you have lots of social media commentary, or whatever it might be, it becomes much easier to do. You can almost do quantitative analysis of the qualitative data. It's just a very … the methods change. Or at least more methods become available to you.

Chip Koziara
Yeah. One of the things I wanted to add is I think a lot of bigger companies are launching new products, new services, things like that. I think that's where a lot of that customer interview and customer feedback becomes important, even though the company is big. And so I think it also depends contextually where you are within a company. So like Uber Eats. When Uber Eats launched years ago, Uber was already a big company at that point and they launched this new thing. And so how Uber Eats functioned back then is different than how it would function today, or Netflix when they first started offering their streaming service versus their DVD service, same type of thing.

Stuart Balcombe
Right. Yeah, I think that's a great point that it's all contextual and it's not a … I think sort of a big takeaway here is that it's not like a one-and-done-type process, right? These things, as the data changes as the market changes, sort of something to be constantly reevaluating and new questions come up, you need new ways to answer those questions.
Cool. This has been really awesome. I know we could go on for hours and go really deep on any one of these things that we've talked about. But for somebody who is just starting out and maybe hasn't taken a data-driven or done a deep-dive into their data, or any available data before, what's the starting point that you would recommend to them? There's a lot of noise out there. What would be the one place that you'd recommend they start?

Chip Koziara
Yeah, it's a great question. My POV is that people try to optimize too quickly. And so people jump immediately to, "Let's pull out this shiny, new machine learning library." Which is super cool and super interesting. But I think for most people starting out, I think the best thing to do is to fire up SQL, pull a query off of a database, and just get the raw data, and look at the raw data. And then pump that into something like Excel or Google Sheets, and really just start there.
And I think getting more complicated where maybe the SQL report that you're running, you add some CASE whens to start to clean the data and add a little bit more structure. Or even hacking that in Sheets where you're doing different string matching or regular expressions to try to clean the data too. I think just starting with super simple tools to dig in and see what your users are searching for and what your users are trying to achieve, and starting there before trying to get too crazy.

And I think once you do get to that point where you want to start to optimize, I think Python has a lot of different, interesting libraries. R does too. I'm partial to R. But it's also a pretty ugly language and Python is probably a better choice for most folks.

Stuart Balcombe
Cool. Well, this has been great. Where should people go if they want to keep up with what you're working at at Eats, or just generally? I know you write as well. Where should people go to to keep up with you?

Chip Koziara
Yeah, thank you for asking. I'm excited to get that plug in too. So I am Chip Koziara on Twitter, and then also just chipkoziara.com. Those are the two best places to follow me. I also write a newsletter which you can simply subscribe to on my website.

Stuart Balcombe
Cool. Well, thanks so much for doing this. This was awesome.

Chip Koziara
Thanks so much for having me on, Stuart. I really appreciate it.

New episodes released every week

Subscribe and get every new episode in your inbox.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
No spam, guaranteed. 😁