How PandaDoc Manages 800 Product Updates Per Month on the Path to $100M+ ARR (Interview with CTO and Co-Founder Serge Barysiuk)

Introduction: Serge Barysiuk and PandaDoc's CI/CD Approach

Serge Barysiuk

00:00Hi, everyone. My name is Serge Barysiuk, and I'm CTO and cofounder of pandadoc. And today, we're gonna be talking about how we are doing this, continuous delivery of the product. My talk is gonna be slightly different from the business talks because I'm gonna be talking more about the production side. That's what I do. And I manage product and engineering, so that's why sometimes I have arguments with myself. Yeah. But so 800 product updates. You might call

The 800 Product Updates Per Month Claim

Serge Barysiuk

00:30BS, but I checked today, like how many did we do today? 24 so far. Day is still going. And, so that's all the updates we're delivering to production, which our customers can can already use. The question is, why would you care about this? And so let's take a look at the product life cycle. And my assumption here and hunch that many of you founders and even ourselves, we are playing in these two first stages, introduction and

01:00growth. Right? What's important in those two stages is that

Why Delivery Speed Matters: The Product Life Cycle

Serge Barysiuk

01:07so we need to learn a lot and iterate to in the first stage, to find this product market fit, to test ideas, to validate things. And with the growth stage, so we need to close this gap really quickly to deliver the value to the customers and actually get this the leader position on the market. Right? What do we do to do this is we're going through this continuous learning cycle. Right? So we have a hypothesis, so

The Continuous Learning Cycle and Its Bottlenecks

Serge Barysiuk

01:38we do something, we launch this, we test and validate, we measure, then we iterate and change. The speed of this continuous learning cycle depends on the speed of the slowest link in the cycle. And the problem sometimes problems, they're in the different stages, but most of the problems you see, they lie between the implementation and review. And the reason why that so your idea in initiative is not just only one in your company. And when you're

02:10ready to push this out to go and test and validate, so what you face is tons of things is happening in the product, and you need to integrate this in the code, and you need to merge. You need to talk to all the, developers, product managers, make sure that it's all integrated. But then you also need to see together, package the whole thing, deliver this to production, and so you need to go through the testing. You

02:32need to go through the deployment and all this stuff. Right? What it does essentially, it actually slows down this learning cycle because you don't know if it's gonna stick or not, and you need to iterate, you need to do this quickly. So that's why people came up with this concept of continuous integration and continuous delivery, and, it's also known as CICD. You've probably heard about this. Right? And in many, many cases, people actually talk about more

02:59of engineering side of the story. There are other sides of the story. And we'll sprint through engineering side really, really quickly just to explain how it works, and we'll focus more on the product and other sides. So with the engineering, the way how it works is the whole concept is so this is courtesy of GetLat and how they explain continuous integration and continuous delivery that so when we start working on something, so we just create a

CI/CD Explained: Engineering Side

Serge Barysiuk

03:23branch, we're continuously delivering those little updates, to the master branch, which is the latest version of the application. But when you do this, it goes through the pipeline and it automatically tests and, do all the steps needed in order to integrate it. So you make sure that it's not gonna break anything, it's not gonna cause any issues, etcetera, etcetera. And what I mean by this is those steps in pipeline, it's testing automated testing, end to end

03:50test, security, packaging linters, code style, and all of this stuff. And, the next stage is the continuous deployment. So what is that? It's like you need to package also your app in order to deliver to production, containers, libraries, updates, whatever whatever you need to do this, also automatically. So what it helps you to it helps you to eliminate this bottleneck on the engineering side to launch this really quickly and accelerate the cycle. This is a real

04:17life example of pandadoc, one of the pipelines, one of the services. So what would you do? We automate all the steps, security, end to end tasks, packaging linters, code styles, and everything. So when you do this and when you merge, like, goes through this, pipeline, and you're kind of sure that it's not gonna break anything. So couple of things what we learned so far, and if I would give just one advice to you guys, if you're

04:45interested, start doing this as early as possible because it is much easier to do this in the very beginning than later apply to to everything. Because if you have, like, 200 services and you need to apply this to all the microservice and all this other, just such a big project, to to take that, sometimes people are hesitant when to start. And a couple things, but the modular architecture, loosely coupled system will definitely help because it will,

The Pipeline in Practice at PandaDoc

Serge Barysiuk

05:11again, decouple the whole thing, and you can independently, launch and update things, in the app. Now let's move into the product side of the story. And the the product side of the story sometimes is overlooked because, I mean, engineering will figure this out, and they're gonna be launching this, but, it's actually even, I think, more important than on the engineering side.

05:37Very, very high level, those are type of updates you are producing, in your, probably, day to day jobs. So to update the product, so you're fixing some bugs, so you're doing some technical speed improvements, you're doing some UX improvements, you're up to launch some features and all these things. So the first two is like you probably deliver to the customers as soon as possible because you wanna, you know, squash this bug and move on. Right? The

06:02latter two, they are more complex because sometimes you don't know what you're doing and you need to test. And again, that's where you apply this continuous learning cycle. That's what we learn. Right? So the product manager is have a hypothesis, so we're gonna test this. We're gonna launch this. We're gonna measure and how it's gonna affect everything. What I find very often in the mindset, we're constantly thinking about the end stage. Right? Controversial image, I know.

06:35And this is from Agile World, but I think it's actually a good representation the way how you might think about this is like when you think about the feature, you're thinking about this end stage, and you can do continuous integration, continuous delivery all day long, but if in the middle of this process your customers can't use it, you're not learning anything. If you are waiting like you're doing continuous delivery every day, an update, but the customers

Advice: Start CI/CD as Early as Possible

Serge Barysiuk

06:58can't use it for the next three months, it doesn't help you to learn. Right? And so the mindset should be changed in order to switch to this, you know, learn by iterating approach. And sometimes it's a little bit hard because sometimes when you do an iteration and then you need to take two steps back, remove something, rework something, this waste is create this, you know, like tensions. Oh, I wanna do this, but sometimes it's needed. So

07:25and I think it's a

07:28mindset shift should should happen, like, on the product management side. Now imagine everything is working great. So your developers delivering this code continuously. You, our product managers, are thinking about this in iterations, like slicing and dicing this correctly. And so you have this amazing update, and it went live through the continuous delivery. Do you wanna show this skate to 100,000 customers altogether? Probably not. Right? So you're not ready for prime time, so you need to test

07:59this thing first with a very, very limited subset of the customers. Maybe get their feedback, maybe iterate. When you're ready for prime time, then have your prime time. And in order to manage this scale, people came up with the idea of the feature flag. So what is feature flag? Feature flag is basically a toggle which tells you this. If this thing, update, change, feature UX update, is visible to a particular customer. And in order to manage

Product Side: Feature Flags and Gradual Rollout

Serge Barysiuk

08:29this on scale, not in the code, there are different systems which allows you to do this and define those feature flags and independently manage them from the code. So when you write your feature, you wrap this in a feature flag, and there is a system. So this is split IO real system, and you can use it to manage those tags and gradually roll out the features to your customers when you need it, like to learn, to

08:54get the feedback, to see behavioral metrics, and all that stuff. So the pandadoc, we probably right now have roughly 300 feature flags, like, in the different stages. Some of them, they're rolled out completely. Some of them, they're, like, still, maybe like one, two percent, but this is how it works. And the product managers can actually do these changes independently from the system. They don't need to release anything, so it's independent. And if something goes wrong, they

09:21also can go down to zero, and nobody will see this.

Rollout Strategy: Internal to General Availability

Serge Barysiuk

09:26Our strategy about the audience and the rollout strategy is kind of like very, very slow in the very beginning, so we take this only internally. We test this, so we have teams who are using pandadoc every day so they can give feedback. We're gradually rolling out this to other users. We like to work with the customers we know and we trust, so it's a CAP, customer advisory board, some early adopters, some recruited users who might be

09:52interested in this feature. But then we are ready for prime time, so we start testing on a little bit bigger audience. The tendency is and a strategy we like to use is lower risk to higher risk because you probably have also diversity of the customers in the very beginning, so you might go with the maybe, I don't know, like a free product customers or something, and then switch to more bigger customers, more value customers, and all

10:16this stuff. Following the well known patterns, early access, alpha, beta, and then general availability.

Recruiting Early Adopters

Serge Barysiuk

10:25Now, how do we recruit those early adopters? You'll be surprised, but actually people like to try new things, especially if they're excited about those new things. And the way how we do this is there are multiple places where you can grab them. First is the community boards. So we use user voice or some other things, it doesn't matter, where we collect the feedback from the customers. If we see, I don't know, like a particular feature and

10:49we start working on them, we have a list of the customers we can contact, and if we have the first version ready or something, or even to talk to them so we can can invite them and try. Customer advisory board, which we have also, support, surprisingly, is a good source because people are asking about something. Hey, do you have this and that? You might have, but it might not be released to the customers yet, and you

11:12can enable to get the feedback and other things.

11:17So now you launch this. Like, you're continuously delivering all the stuff. In order to complete the cycle, what you need to do is also you need to measure because if you want measure, you can improve. You don't know what to do next. You can assume, but in reality, you don't know. So our strategy, how we do this, obviously, different depending on the stage. And in the very, very beginning, so when we launch this, when it's kind

Measuring What You Launch: Amplitude and Tableau

Serge Barysiuk

11:44of like a very early stage of the feature or like a change, so we try to do this a little bit hacky, more like a behavioral and usage type of thing. We use Amplitude, and Amplitude is the analytical platform where you can apply on top so the product managers can do this and understand, oh, people even interact with this or not. Right? And, when it's a more mature stage of the feature, so we try to apply

12:09more structured approach. So we have a TARS dashboards for the feature, so we move this into, Tableau and our own, kind of the analytical platform to have a deeper look, especially if the feature affects the revenue.

12:25Obviously, qualitative feedback. It is very important, especially in the very beginning because sometimes there's behavioral things in the user, they don't show anything because if they don't use, you will not see anything in the analytics side, and so you need to talk to users as well. It would be end of the story, and your developers would deliver this to customers. Your product managers would iterate and find this, you know, ideal, feature for the customers and tune

The Challenge of 40 Teams and 300 Feature Flags

Serge Barysiuk

12:58this for the customer, but life is not that easy. And the the life is not easy because when you hit scale and when you have 40 teams working on the different things in the different parts of the application, launching this to the users and 300 feature tags, what it creates? It creates a big mass which you put on your customers. They see this. They this is new. This changed. This has changed, you know, like in word

13:25testing and and validating. That's probably not what you want to do. Right? Because you wanna satisfy your customers, and that's why you are actually learning a lot what to do. So, therefore, you need some a layer of policing of what you do. And the way how we handle this at pandadoc is with the product market and and the packaged product launches. So what is product launch? So think about this as a story and, you know, like

Product Marketing: Packaging Releases into Launches

Serge Barysiuk

13:52a topic you are working on. It can be improvement, can be speed improvements, can be a feature, it can be, something you decided to improve in the

14:01customer pain points, some other things. Right? And so what the product marketing does is it creates a story, and you start packaging those dispersed features across the system you were testing into something more tangible and connected as well, and which is enabled with the proper launch, blog post, pictures, and also all other teams are enabled as well, sales, customer success support, because sometimes we overlook the thing. And so we launch this and that, feature flag to

14:32100%, it's done, but it's not. And the teams don't know about this. They can't sell it. They don't know how it works and all these things. So this is very important to kind of close the loop and capitalize on what you've been working on because you invested a lot of resources. And so that's what we do when we package them, and that's where you might see, I don't know, like a blog post with the announcement of

14:55some module, I don't know, application or improvements in the speed and stability. That's that's a product launch. To summarize all the things, engineers are continuously working on this and continuously integrating the code. So it's a low risk. We don't create those bottlenecks in the integration step and in the delivery step. Everything is rolling out continuously. The product managers, they are using feature flag testing this, sometimes kill things, sometimes, finalize and, polish, and the product marketing is

15:26packaging them into something presentable to the users to capitalize on your investment. And that's how we do this at pandadoc.