How Mode Analytics Used Data to Find Product-Market Fit (Talk by Co-Founder Benn Stancil)

Introduction: Mode and the Talk's Theme

Benn Stancil

00:00All right. Let's see if this works. Okay. Cool. So I'm Ben. I'm gonna talk about finding product market fit with data. A quick bit about myself. So I if this sort of talk is a lot of stuff drawn from a blog. This is kind of the vibe of that blog. But kind of more importantly for this talk, I am one of the founders of a company called Mode. Mode is a BI tool built for data teams.

Mode's Early Product and Customers

Benn Stancil

00:26It looks like this. It also looks like this. This is kind of the visualization, draggy droppy BI stuff. And here are some of our customers. So these are some folks that use the product today. But obviously, this is not like what Mode was like in the early days. This has taken some time for us to build. Back in the early days, this was actually Mode's first product. It didn't look like all that fancy stuff. It looked

00:49like this. And instead of having any customers, we had this. These were the other two cofounders of Mode, one of whom is here and talking in like forty minutes later. So check them out. But anyway, so the point here was generally where Mode is today obviously is not where we started, and there were some rough moments along the way that took us some time to get there. And so this is actually a message from one of

The Product-Market Fit Problem at Mode

Benn Stancil

01:13his other co founders back when things weren't doing so great that said, hey, you know what? I don't actually think things are looking so good. I don't think that Mode has product market fit right now. And so this was like what the world was like back before some of the stuff that we built and some of these customers we had. And this was like kind of that world. So this is kind of a very basic graph

01:36of what product market fit is like. Prior to product market fit, everything sucks. You get a message like this from Josh. After product market fit, everything seems good. And so the question really to talk about here is how did things get better? Like what is it that got us from the left side of that graph to the right side of that graph, and how do we actually figure out where we were before to get to where

01:56we are now? So I want talk about two main reasons and two kind of big things to think about as you're trying to do this. One of those reasons is figuring out what people like. This is kind of obvious, but I want to talk about some ways that you can do this a little bit. And second is figuring out when to actually start scaling, that finding product market fit isn't just a matter of saying, hey, what

Finding Traction: The Knot Case Study

Benn Stancil

02:14do we need to build, but it's also timing it right. And so I want to talk about that a little bit too. So to get things started, we can talk about this finding traction bit. And to talk about that, actually, I'm not gonna talk about Mode. I'm gonna talk about a different company that y'all are all probably familiar with but aren't necessarily aware that you're familiar with, which is a company called The Knot. So they actually

02:35had an office, I think, in this building for some period of time. But what The Knot is, is it's a website for building websites for your wedding. So if you've got a friend get married and you go to websites like this, like danielle.loveshaaron.com or whatever, that are like your friend's wedding websites. The Knot is the vendor that hosts these, and so when people want to be able to set up these websites, they can go to the

02:58Knot. It helps them set up things like registries. It helps them set up, you know, here's how to RSVP for all these things, that kind of stuff, and the Knot is the service that does that. And so the Knot was one of Mode's early customers. When they first started using Mode, they had actually recently just released a mobile app. So they had this kind of main website builder, but they also wanted to have a mobile app

03:18for people to be able to do the various things they need to do to plan their wedding and that kind of stuff. That was this app, the one hundred and thirty third most popular app in the Lifestyle section of the App Store.

03:30When they released it, it didn't do so hot. It was something that was not terribly popular. It was a really important initiative for them to figure out, hey, this is what we think of as the future of our business. We need to be able to figure out how to get people to do this sort of stuff on mobile as well as on the website, and so how do we build that? And this app did not have

03:48sort of product market fit when they initially launched it. Their first set of experiments were basically like build a bunch of features, go look at how they're used on something like Google Analytics, see if they work, if they don't, try again, and kind of like rinse and repeat. That it was basically just iterative process of build features, see what happens, build features, see what happens, build features, see what happens. It didn't really go anywhere. The app

04:09basically stayed stuck where it was, and it never actually got popular. And so what ended up happening was one of the folks who was the head of their data team started poking around how people were actually using the existing app. And he made this chart. This was a chart that he made in Mode. The specifics here kind of don't matter that much, but basically you can imagine each one of these further out rings is people sort

04:30of progressing through different stages of the website. So the green bar might represent the homepage, the purple bar might represent the registry page, whatever, and so it sort of tracks the different patterns of how people are moving through the site. And so he made this chart to kind of see how this behavior looked, and the thing that he noticed was this down here, which was this very kind of strange brown spike of like, hey, why are

04:52people continuing to come back to the same page over and over and over again? They're consistently doing it. Like, what's the deal with what's going on here? Seems like there might be some pattern here to pay attention to. And what it turns out is it was this page. It was a page that was like a countdown for your how long until your wedding. That it was a single page that said, hey, you're gonna get married in

The Knot's Mobile App and the Countdown Page Discovery

Benn Stancil

05:11eighty six days, in eighty five days, in eighty four days. And people kept coming back to it day in and day out, like taking screenshots of it, and then sending it to their friends or posting on social media or whatever. This was a kind of small thing that they built. They didn't think it really mattered when they first built the app, but in putting it together they realized this was how people were actually using it. This

05:29is the thing that people wanted to do. By looking at that chart and kind of following the behavior of what users were doing, they actually started leaning into this behavior and built out what is this, which is where it looks better, you can add these photos, it was easy to share on social media, and actually the app started to take off because of it. They really leaned into the way that people were using it rather than

05:47trying to force them to do the things they originally wanted them to do. Another example of this is a company called Bourbon. This may be an example some people are familiar with. This was like Foursquare for bourbon. You would go somewhere, you would drink bourbon, you would take a picture of it, would check-in, you say I had this bourbon, I liked it. Turns out nobody really wanted to do that.

Instagram and Latent Behavior Patterns

Benn Stancil

06:08The people who were doing this actually didn't really care about the check-in stuff. They just like take pictures of their bourbon, and so the app was like, hey, wait a minute. Maybe we'll just have more picture sharing stuff instead of bourbon check-in stuff, so they changed their name from Bourbon to Instagram, and that's now how Instagram actually got started. Again, this was not the point of the original product. The point of the product was to, I

06:29guess, have social networks around Bourbon, but they saw this other kind of latent behavior that they followed. And so what ends up happening a lot of times in early stages of products is something like this, where you'll spend a whole bunch of time building these features, and you'll have some small gimmick in the corner, and the small gimmick in the corner is the thing that people are actually excited about. And so when you're looking for product

06:47market fit, this is probably the best thing you can do, is pay attention to what are your girls in the red dress that everybody's actually distracted by when you're trying to get them to build the thing or do the things you want them to do.

Why Data Is the Only Way to See the Path

Benn Stancil

07:01The part of this, though, that's kind of important is that, and people have probably all seen this. This is basically the same as this, of like, okay, can design a thing and people will do something else. The thing is, like in SaaS apps, there is no actual physical path that you can see. You can't actually see like the worn down grass on the dirt. You have to be able to track this in some other way. This

07:21is something that only sort of materializes in your data, it only shows up in stuff like this. And so really, the job of us, as early stage founders, as early stage product folks, is to pay a lot of attention to this, to try to do the best we can to kind of recreate that GRAS pattern in the data that we have and the behaviors that we see, to be able to see what are those places that

07:41people are actually cutting the shortcuts in the path. And the more that you do this, the more you pay attention to this, the faster you can find these paths, and that's a really big part of what this job here is, is kind of more than anything, as Andrew Chen says, who's guy who talks a lot about product market fit stuff, really the job here is to reduce the time to do this. Eventually you will see these

07:59things. If you build the app long enough, you build your products long enough, some things will emerge that are the kinds of behaviors that people follow. Somebody will just tell you them, but the more you look at the data and kind of the more you're looking for these anomalies, the faster you'll be able to find this, and the better off you'll be. And there's a reason that speed matters, which is there's actually sort of another line

08:18on this chart. This isn't the only line that really matters. There's also a line here of how much money you have left as a business, and so if this takes too long, this money line basically goes to zero. Also, if you start to try to scale the business too quickly, this money line will go to zero a lot faster, which kinda leads to the second problem of, okay, once you sort of find these interesting patterns, once

08:38you find these things that might be places of sort of markers of product market fit, how do you decide actually when to scale? And for some people, actually, this is the bigger problem in finding the interesting kind of elements of what people like. So this is one of the famous growth VPs from Facebook who basically said, actually the bigger problem that startups have isn't necessarily finding these things, but it's identifying when they have it, rather than

Mode's Own Premature Marketing Spend

Benn Stancil

09:00like, hey, we had early signs of something. We think that's good enough. We're ready to scale when you're actually not where you think you are. So to use another kind of Mode example, this is an actual graph of our marketing spend sort of anonymized by month and with some numbers wiped out, but you can all kind of see the problem here, which is this giant green mountain of wasted money. This was also the time that Josh

09:23was like, hey, things don't look so good right now, and basically this was us thinking we'd found product market fit and we hadn't, that we thought we had found something that worked, we thought it was time to scale, to spend more money on ads and all that kind of stuff, and it turns out, kind of obviously, we didn't, and so then our money left line, if we draw that over this, would look like this. The zero

09:43is not the same, like zero is down here, but still, that's not the chart that you want to see. So the question then is, okay, how do you know when you're ready? How do you actually know that it's time to start spending more on marketing or scaling the product as opposed to kind of continuing to develop and iterate. My initial answer to this, probably around the time that we started spending all that money on marketing, was

Why Simple Retention Metrics Can Mislead

Benn Stancil

10:07like it's probably just retention. If people keep using your product, it's probably good. That's good enough, right? That's what other people say. So people have probably seen charts like this. This is from Mixpanel, basically, like the number of people who come back after a certain number of days. If this number looks good, then great, you're ready to scale. If it doesn't, don't. Simple enough, right? So not really. So the example I wanna call out, like why

Greenhouse vs Front: Different Fit Metrics

Benn Stancil

10:30this doesn't actually work, I'm gonna draw a couple examples from two companies that y'all may have heard of. One is Greenhouse and the other is Front. So Greenhouse is an applicant tracking system. Basically, if you have job recs on your website, Greenhouse will power those. They will then help recruiters of usher candidates through the job process, so they'll move people from one stage to the other. Candidate goes from an interview to an on-site to an offer,

10:52things like that. And so it's a management system for that and for recruiters. Front is a shared inbox. It's basically for support teams to have a single inbox where people can email, and multiple people for the company can then respond to that email and see the conversations that everybody's having. And so the next few slides, I'm gonna talk about these things using a graph that looks like this, which looks like a huge mess right now, but

11:16I can explain what it means in a much simpler version, which is this. So basically this is looking at users and the days that they used a product. And so if you look at this first user, for instance, what this is saying is the user used the product on day one, on day two, and on day four, or on day two you can say that users one, three, four, five all used the product, but number two

11:35didn't. Or if you look at an individual day, can say, hey, this block represents user one using the product on day two. And what you can pretty obviously calculate from this is retention rates. So on day one, all five of them used it, so it's 100%. Four out of five use it on day two, so it's 80% kind of and so on out, where it's like eighty, sixty, sixty, forty. So basically if you then zoom back

11:57out of this, you can get a chart like this that shows the different patterns of how people are actually using the product. And so this is a hypothetical. It's like me making it up of what Greenhouse may well look like. I don't actually know, but you can imagine a pattern that looks like this for Greenhouse, where people use it sporadically, like recruiters aren't logging in every day, people aren't progressing necessarily through recruiting pipelines every day. Some

12:20of these people might be hiring managers that are only coming in when they actually have interviews, things like that. So they have this kind of sporadic pattern of returning. But on this chart, if you look ninety days after, like three months into this, actually 97% of these users still come back, so in the time between ninety and one hundred and twenty days, there's actually 97% of these 200 users are all still engaging with Greenhouse, which seems

12:43pretty good.

12:45So this is Front. This is another version of this chart. Could represent the same thing. You can imagine these blocks mean the same thing. In Front's case, it's very much intended to be a daily usage app. It's something that's supposed to be like you use it every day. It's like your email. If you use your email every day, or most people probably use their email every day, they aren't using it once a week or something like

13:02that. So in this case, though, if you draw the same line because of this usage pattern, after three months only twelve percent of these people are actually still using From. So it's a very different sort of retention pattern, a very different group of people who are using it, and in this case you'd probably look at this and say like, that doesn't look so good. So looking at these two charts, on this top one from Greenhouse, you'd

13:21like, okay great, now it's time to expand, it's time to scale, let's go hire more salespeople, let's jack up that marketing spend chart, all that stuff. We have product market fit, it's all great. For this chart, you'd be like, no, we're not there at all. We have some few people who like it. We've sort of found maybe the path in the sand or the path in the grass that people are following, that a handful of these

13:42people really gravitate towards, but clearly this is a product that does not seem terribly sticky yet and does not have product market fit. The problem here is if you just look at daily retention rates for these two things, they will look exactly the same. The way this math works out is these two things can actually show the exact same retention rates on a day by day basis. And really what that means is that metric, even if

14:04that metric seems like a really good one, it hides all of these other patterns that may not actually show up in simple metrics like retention. And so the point here is what Steve Blank says. Steve Blank is author of The Epiphany, which is one of the canon of Silicon Valley kind of stuff. Basically, these sort of one size fits all approaches of just use retention or just use certain metrics do not work for all startups, you

Defining the Right Fit Metric for Your Product

Benn Stancil

14:28have to really think about what it is that your startup needs. And so what do you do? Instead of that, basically define what fit looks like for your very specific product. What does it mean for someone use your product in the way that you intend them to use it, or in the way they should be using it, and how would that actually show up in these sorts of retention metrics? So for instance, for Greenhouse, that may

14:48not look like just this three month retention or daily retention. It may actually be things like how many hires do they make? This is actually, so both Greenhouse, the reason I use Greenhouse and Front for these two things is they actually published a study a bit back with one of these VCs, saying hey, these are the metrics they use to decide when to scale, and this is actually what they said. So it was like okay, if

15:08we have the number of hires made, if that's staying steady, that's product market fit for us, that's a sign that we're ready to go. For Front, they want users to be engaged early and often. Again, it's not a sporadically used app. It's an app you're supposed to use every day. The point here is to use it early and often, and so their definition of product market fit, their things that they looked at to see when it

15:26was time to scale, was how often are people sending messages in the first thirty days. That gets them past the tire kicking phase of them logging on and sending a few messages in the first couple days, shows that they're kind of wanting to stick around, but it doesn't have to wait for months or years to see if they actually use it, so this was the metric they used. And so if you're thinking about these things, you're

15:43always trying to say, okay, what's the metric for my business? How do I actually do this? That kind helped brainstorm a handful of ideas from other businesses or people who've thought about this. One, if you are a product that needs to be used every day, something like What Front Is makes sense. Things like DAUs over MAUs, which effectively measures how many days in a month people are using your product, is really important. A high number there

16:05matters. So 50% are using it say fifteen days a week, or fifteen days a month is a really important kind of threshold. Similarly, if you're like a SaaS business that's selling B2B, obviously in that case it usually follows much more of like a work week kind of pattern, and so if people are using your product more than four days a week, like how many percentage of your users are using it four days a week shows that

16:24hey, these are people for whom the product is really sticky, it's something that's really valuable. If you are a product that has a lot of competition, then something like a high NPS is a pretty good sign of product market fit, because people have a lot of choices. You need to know that people actually like your product. It's a thing that they want to use. It's a thing they will choose over the other alternatives they have. NPS

16:43can actually be a pretty decent measure of that. If your product is a vitamin and not a pain killer, so if your product is something that's kind of a nice to have, you want to actually see that people will be upset if they take it away. So this is actually a metric that I think Superhuman used, the email client, where obviously people have email clients. That's very much a vitamin, not a painkiller to have that, so

17:02they wanted to check and see if people would be disappointed once they started using it if you take it away. Will I be upset about that? And if they're not, then people could probably walk away from it pretty easily. And the last one, if you're a product that's sort of an unknown demand, say you're trying something new, you have something that you don't actually know that people pay money for, in this case you actually should see

Product-Market Fit Is an Ongoing Journey

Benn Stancil

17:20them paying money. You want to see things like revenue growth. You want to see things like LTV over CAC, stuff like that, that demonstrates people actually put money where their mouth is. They're not just going to say, yeah, it looks great. We'll kind of keep using it. We'll actually pay for this and find real value in it. So the last point I want to make on before kind of ending things with questions is this chart. I've

17:41shown this chart a number of times. I will say this chart is actually a lie. This is not what product market fit looks like at all. It's a nice little graphic or whatever, but in reality, as you're building businesses, product market fit looks much more like this, where it's this kind of iterative thing. You find it, and then you move to a different market. You try to market to a new segment. You expand to selling to

18:00different types of customers, and you no longer have product market fit with those folks, and you have to find it again. So it's like a sort of ongoing journey to do this. This is not something where it's like you are pre- post product market fit forever. You are pre- and post product market fit for particular types of buyers, for particular products, for particular segments, and all those sorts of things. So these two things, it's not actually

Closing Remarks and Q&A

Benn Stancil

18:22like these are the two things you have to figure out, is what people like and when to scale. It's actually you have to constantly be figuring out what it is that people will like next, the people that you're trying to sell to, what's important to them next, and figure out when to scale further. So how do you keep moving up that chart when you're ready to move to a new market, to a new product, to a

18:40new segment, all those sorts of things. It's an ongoing part of continuing to build a startup. So that's a wrap. If you want anything more from me, this is the Twitter and the Substack, and you can also just search for my name on LinkedIn. It'll show up, or you can go to benn.work. That's it. That doesn't say questions. Cool. I'll stop there, and if there are questions, I can answer questions.

Nathan Latka

19:07Alright. Great. Thank you.