800 Updates a Month, 300 Feature Flags: How PandaDoc Ships Continuously
Serge Barysiuk says PandaDoc delivers roughly 800 product updates to production a month across 40 teams. The CI pipeline is the part he sprints through; the flags, the rollout ladder and the packaged launch are the rest.
On this page
Serge Barysiuk opened with the number he knew nobody would believe, and then went and checked the counter. “So 800 product updates. You might call BS, but I checked today, like, how many did we do today? 24 so far. Day is still going.” Those are not commits or tickets: “that’s all the updates we’re delivering to production, which our customers can already use.”
PandaDoc’s CTO and co-founder said up front that his talk would be different from the business talks — he runs product and engineering, “so that’s why sometimes I have arguments with myself” — and then gave the continuous-integration pipeline that produces the 800 the first third of his fifteen minutes, promising to “sprint through engineering side really, really quickly” so he could spend the rest on everything bolted on top of it. That ratio is the argument. (The period is the talk’s: he says 800 updates, the title says a month, and the day’s 24 is what he offers as the check.)
Shipping speed is not the constraint; the learning cycle is. Barysiuk’s frame is a loop — hypothesis, launch, measure, iterate — and his rule for it is that “the speed of this continuous learning cycle depends on the speed of the slowest link in the cycle.” CI/CD removes one link. If the other links stay slow, 800 updates a month buys you nothing but 800 updates a month.
The engineering half, sprinted through
Barysiuk deals with the part most CI/CD talks are entirely about in four minutes, because he thinks it is the solved half. Work goes onto a branch, continuously merges into master, and every merge runs a pipeline before it is allowed to matter.
- Automated and end-to-end tests — the gate that lets a developer merge without wondering what else they just broke.
- Security, linters and code style — run as pipeline steps rather than as review comments.
- Packaging and deployment — containers, libraries and updates assembled and pushed automatically, so delivery is not a separate project.
His one piece of advice, if he could only give one, was about timing rather than tooling: start as early as possible, because it is far easier at the beginning than as a retrofit. “If you have, like, 200 services and you need to apply this to all the microservice,” he said, it becomes a big project of its own, which is why founders hesitate over when to start. The architecture makes the difference too — modular and loosely coupled means pieces can be launched and updated independently.
The bottleneck he is engineering around is specific. Problems appear at every stage of the loop, but “most of the problems you see, they lie between the implementation and review” — the moment your change is ready and you discover how much else has happened in the product since you started, and you have to integrate, merge, talk to every developer and product manager, package the thing and get it through testing and deployment before anyone can tell you whether the idea was any good.
300 feature flags, held by product managers
Getting an update into production is not the same as putting it in front of customers, and Barysiuk treats the gap between the two as the actual product surface. Do you want to show a fresh change to your whole customer base at once? “Probably not. Right? So you’re not ready for prime time, so you need to test this thing first with a very, very limited subset of the customers.”
PandaDoc manages that with feature flags — a toggle that decides whether a given change is visible to a given customer — kept in a system outside the code, in their case Split. At the time of the talk the company was running roughly 300 of them at once, in different stages: some rolled out completely, some, he said, “still, maybe, like, one, two percent.” The point of holding them outside the code is who gets to move them.
“The product managers can actually do these changes independently from the system. They don’t need to release anything, so it’s independent. And if something goes wrong, they also can go down to zero, and nobody will see this.”
Serge Barysiuk, CTO and co-founder, PandaDoc
That is the mindset shift he says has to happen on the product-management side, and he was honest about why it is uncomfortable. Iterating in public means sometimes taking two steps back, removing something, reworking it, and that visible waste “creates this, you know, like, tensions.” His answer is that the waste is the price of learning at all, and the alternative — holding a feature until it is finished — only feels tidier.
The rollout ladder, lowest risk first
The order in which a flag opens is deliberate, and it runs from the audience that can least be damaged by a mistake to the one that can most.
- Internal first. PandaDoc teams use PandaDoc every day, so the first cohort is the company itself, and the feedback comes back the same day.
- Customers you know. The customer advisory board, some early adopters, and users recruited specifically because they asked for the thing being built.
- A bigger, lower-risk audience. When it is closer to prime time, a wider group chosen from the less risky end of the customer mix — free-product customers before bigger, higher-value ones.
- The public ladder. Early access, alpha, beta, then general availability — the well-known pattern, entered only after the first three rungs have done their work.
Recruiting the people on rungs two and three turns out to be easy, because “people like to try new things, especially if they’re excited about those new things.” He named three places PandaDoc finds them.
Feature requests collected in a tool like UserVoice come with a list of the customers who asked. When the first version exists, there is already a list to contact.
A standing group of customers the company knows and trusts, used for feedback before anything is close to general release.
His surprise entry. People write in asking whether you do this or that — and sometimes you already do, behind a flag, so you switch it on for them and collect the feedback.
Measuring, in two gears
The loop does not close without measurement, and Barysiuk runs two different regimes depending on how mature the change is. Early on it is deliberately rough: behavioural and usage data in Amplitude, where a product manager can see for themselves whether anyone is interacting with the thing at all. Once a feature matures, it moves into structured dashboards in Tableau and PandaDoc’s own analytical platform — “especially if the feature affects the revenue.”
Neither one replaces talking to people. Behavioural analytics only records behaviour that happens, he pointed out, so if users are not using the thing, the dashboards show you nothing at all and you have to go and ask. It is the same discipline PandaDoc’s CEO described to Nathan Latka five years earlier, when he named product quality shipped quarter over quarter as one of the three levers that pulled the company’s monthly logo churn down from five percent, and said you could watch it land in the cohort analysis. This talk is what that looks like once there are 40 teams doing it at once.
Why 40 teams shipping continuously needs a policing layer
And 40 teams is where the tidy version of the story breaks. With that many groups working on different parts of the application, plus 300 feature tags opening at their own pace, what lands on the customer is not a stream of improvements but a stream of change.
“It creates a big mass which you put on your customers. They see this. This is new. This changed. This has changed.”
Serge Barysiuk, CTO and co-founder, PandaDoc
So PandaDoc puts a layer of policing on top, and it is owned by product marketing rather than engineering. Dispersed features that were tested separately get packaged into one story with a real launch behind it — a blog post, images, and, critically, the internal enablement. Barysiuk’s complaint about the alternative is precise: you take the flag to 100%, you call it done, “but it’s not. And the teams don’t know about this. They can’t sell it. They don’t know how it works.” Sales, customer success and support have to be launched to as deliberately as customers are, or the investment stops at the release.
The numbers behind the talk
What the record says, and what the title claims
The talk is dated 17 March 2023, and it carries a headline the tape itself does not support: a path to “$100m+ ARR.” Barysiuk never mentions revenue, growth or ARR at any point — his only numbers are shipping numbers. The GetLatka record for PandaDoc puts the nearest dated revenue row before the talk at $87 million, recorded 21 November 2022. The $100 million row is dated 1 August 2024, more than a year later, which makes the title a forecast at the time it was given rather than a figure.
Two other dated rows give the talk its scale. PandaDoc’s engineering headcount was 277 at 1 January 2023, the closest reading to the talk, inside a total team of 866; by 1 September 2023 the database records 264 engineers and 849 people overall. Set 277 engineers against the 40 teams Barysiuk describes and the arithmetic gives something like seven engineers a team — small enough that the independence he keeps insisting on, in the architecture and in the flags, is the only way 40 of them ship at once without queueing behind each other. The company had raised $106.1 million by then, most recently a $55 million Series C recorded 22 September 2021 at a $1 billion valuation.
The line that decides whether any of it works
Barysiuk’s summary at the end runs the three layers back in order: engineers integrating continuously so integration and delivery are never the bottleneck, product managers using flags to test, kill or polish, and product marketing packaging what survives. But the sentence that governs the whole system came earlier, aimed at anyone about to buy a pipeline and call it a strategy.
“You can do continuous integration, continuous delivery all day long, but if in the middle of this process your customers can’t use it, you’re not learning anything. If you are waiting — like, you’re doing continuous delivery every day, an update — but the customers can’t use it for the next three months, it doesn’t help you to learn.”
Sources — Serge Barysiuk, “How PandaDoc Uses CI to Manage 800 Product Updates Per Month on Path To $100m+ ARR”, a talk recorded 17 March 2023 and posted that October; the available transcript runs to roughly 15 minutes and breaks off mid-sentence in the closing summary, so the final seconds are not quoted here. Mikita Mikado’s earlier interview with Nathan Latka, recorded around the turn of 2017 into 2018, for the churn history. All revenue, headcount and funding figures are dated rows from the GetLatka company record for PandaDoc; the engineers-per-team figure is arithmetic on two of them and is not stated on the tape.
