Metrics Debt

In the same way under-investment in technical rigor can result in long-lasting negative effects on a codebase when left unchecked, under-investment in metrics can lead to long-lasting negative effects in your product organization.

At Quint Growth we work with companies every day to improve how they measure, understand, and act on data. We review many metrics setups and talk to companies about how they approach the problem of understanding and optimizing growth. We find that key questions that companies need answers to in order to make important product and business decisions are often either poorly understood or require quite a significant amount work to answer. This doesn’t vary by company stage, we’ve seen this just as much in small companies as we have in large successful ones. I’ve never spoken to a single company who has complained about their over-investment in metrics, its chronically under-invested in by all post-product-market-fit companies.

There are a number of effects that under-investment in metrics causes:

[Read more...]

Retention Is King

This was originally posted as a guest post on Andrew Chen’s blog here.

There are too many companies asking, “How do we acquire more users?” that should instead be asking “How do we get better at keeping the users we already have?”.

Its easy when approaching the problem of growth to think that you just need to get more users, after all that seems to be the very definition of growth. However, if you take a step back though and think about growth as the maximization of user-weeks over time, it quickly becomes apparent that focusing on retention has a much larger effect than topline growth. This is also much more of a sustainable growth mindset. Rapid user growth followed by rapid user attrition is an indicator of unsustainable growth. Strong retention of users over time is a good indicator of product-market fit, something you’re hopefully looking to achieve anyway.

Viral Factor and Retention

At a high level, retention is more important than virality because if your users don’t stick around they are not able to invite others to your product over an extended period of time. If you have high retention and no virality you will sustainably grow your user-base over time. If you have high virality and no retention you will not. Between these two extremes it gets a bit more complicated. In order to explain in detail we first need to review a couple of terms: viral factor and retention.

It will help as you follow this post to use our in-house growth model to play with the numbers yourself, the graphs we reference later on in this post are derived from it.

Viral Factor
This describes the growth rate of a site or app based on invitations from existing users of the service. This is often called k factor.

i = number of invites sent by each customer
c = conversion rate of those invites (#signups/#invites)
k = i * c

Viral factor on a weekly basis usually looks something like the graph below. This varies for different products, but I’ve seen this shape again and again across the products we consult on. It is front-loaded like this for three reasons:

  1. The effectiveness of onboarding invitation flows.
    Onboarding is one of the few times you have a high level of user attention towards completing a specific goal (signup) and when you instruct users to invite others they often will without thinking much about it.

  2. Users’ level of excitement.
    Humans are most excited by new things, this applies to internet products as well. The excitement users get when trying a new product leads them to share more, but this sharing tapers off as product becomes normal in their everyday lives.

  3. Low invite saturation of users networks.
    When a user first starts using a product they know many more people that don’t use the product than do use it. Over time they share your product with others they know. Eventually, even a user who is very passionate about your product has nobody left to share it with that hasn’t already heard about it leading to lower virality as time goes on. This can also be an issue if your company grows very large, but that is a good problem to have.

Screenshot 2014-05-15 16.55.26

This is the number of users that stick around from one time period to another. There are two ways to express retention, overall retention and week-to-week retention:

  1. Overall Retention
    Overall retention is cumulative over time. If you have 30% overall retention in Week 3 it means that 30% of your users who started at the beginning of Week 1 are still around in Week 3. This is how companies normally express retention when discussing it internally.

  2. Week-to-Week Retention
    For growth purposes its often useful to look at retention on a week-to-week basis instead. Week-to-week retention is how many users move from one week to the next. If Week 2 has 40% overall retention and Week 3 has 30% overall retention than our week-to-week retention from Week 2 to Week 3 is 75%. If week-to-week retention is below 100% it means we’re still losing users.

Retention on a week-to-week basis usually looks something like the curve below. It is lowest from the first week to the second week and approaches 100% as time goes on.

Screenshot 2014-05-15 16.55.20

Why is retention so important?

In order for virality to be more important than retention your viral factor must be greater than your overall retention up to that point in time. We’ll prove this mathematically later in this post. The math is hard to simplify exactly, but there is a basic rule you can follow which approximates it. If you take only one thing away from this post it should be this:

Do not focus on improving virality unless your overall retention is stable, not continuing decrease after some reasonable period of time.

To help illustrate this, lets look at a few examples:

  1. Your product has a very high immediate viral factor.

    If your product is front-loaded with invites that are sent out and accepted at a high enough rate, it is possible to achieve an immediate k > 1 viral factor. In this case, if your invites have an acceptance rate that is quick enough, your monthly active user numbers will continue to grow even if you have zero retention past the first-use of your app/site. Growth of the high virality, low retention type is almost always unsustainable as the viral loops being exploited to attain quick k > 1 virality eventually expire. This has been the downfall of many companies that appeared to grow fast, raised a lot of money on that growth, then quickly died. Like Viddy…


  2. Viddy growth model with viral channels working looks like this.
    Viddy growth model with viral channels broken looks like this.

  3. Your overall retention is high and decreases slowly as time goes on, but you have strong virality.

    In this case, products actually do see long term benefit from focusing on increased virality, but it’s often a false signal. Its only worth focusing on virality if you think you can improve virality more than overall retention. If you switch focus too early it will lead to sub-par growth metrics. This is because the compounding effects of retention improvement are much stronger than those of virality improvement.

    We can illustrate this easily with our retention/virality simulator and setting week-to-week retention and virality to be equal. We can then measure the effect of proportional changes to one of the other in terms of number of users that we have at some point in the future. In the real world virality is not likely to ever be equal to week-to-week retention, but for purposes of this example please disregard that as it helps illustrate our point in the clearest way possible.

    Base Case – Equal Virality and Retention: ~88k users total in Week 7, 44k from retention and 44k from virality as seen in the stacked bar chart.

    20% Increased Week 1 Virality: ~110k users total in Week 7, 53k from retention and 57k from virality.

    20% Increased Week 1 Retention: ~125k users total in Week 7, 65k from retention and 60k from virality.

    As you can see here, changes in retention have long term effects that have a greater effect than equivalent changes in virality.

  4. Your overall retention is high and stable.

    If you have maximized your retention to the point where you think you can increase virality more than overall retention, even accounting for the compounding effects of retention, then it makes sense to focus on virality.

Prove It!

It’s easy to model growth in Excel and there are some great models that have been shared online to help do this. The one I use is available for download at It’s a slightly modified version of this great one from Rahul at Rapportive. It will give you a nice overview of how you can expect to grow if you plug in some numbers. From a user accounting perspective this is great, but its a bit hard to conceptualize how growth actually works by looking at it that way. To get a bit of a different perspective, building a tree to see exactly where users that exist in a given week come from is quite helpful.

In the tree below w0 represents some set of users that start at time 0. Each level of the tree is a week in time. At each subsequent level you get users that stick around from retention or are invited via user virality. The number of users at a given node is just the product of all the nodes leading to that point. The coefficients represent the viral and retention factors over time, v2 is the viral coefficient in week 2 from the above virality graph, r3 is the retention coefficient in week 3 from the above retention graph, etc.

Screenshot 2014-05-13 17.24.01

The number of users at any given level can be simplified into a recursive equation.

Screenshot 2014-05-13 17.15.28

As you can see, this matches exactly what we saw in the tree graph above. The leading viral factor (current virality vi) matters relative to overall retention (trailing product of rn‘s).

Visualizing Retention and Virality

For the client work that we do at Quint Growth we built a tool to plug in numbers and visualize the retention/virality tree. Its the same tool we mentioned in the beginning of the post and used in Viddy example above. We’ve found it incredibly useful for visualizing how much more important retention is than virality (and the few cases in which it is not). We’ve made it available at and hope you find it as useful as we do!

Special thanks to Isaac Hodes for help with the d3 visualizations for the retention/virality visualizer.

Add Funnel Steps to Improve Conversion

In my last post I discussed how an understanding of consumer psychology can be extremely helpful for understanding how to design funnels that convert. In this post I’m going to give a concrete example of how I did this at Lookcraft to drive 18% conversion through a funnel with 18 discrete questions and 4 major steps. I got 18% of users who hit our homepage to enter their email and password, information about their fashion taste, and full sizing information. I’ve used these same tactics across multiple different products to similar effect with other companies I’ve worked with or consulted for and hope they will be useful for you as well.

[Read more...]

A/B Testing is Expensive

Over the last few years the startup community has really gotten behind A/B testing and hyped it up quite a bit. There is a more nuanced point about the downsides of A/B testing that needs to be understood: A/B tests are very very expensive for most startups at the time when they matter most, early in their formation.

[Read more...]

Goodbye Wholesale Brands

The next few years of e-commerce growth are going to fundamentally change the way soft-goods retail works today.


E-commerce has been growing rapidly for the last 6 years with equally rapid projected future growth. To date the growth has been mostly powered by existing brands moving rapidly online. Great direct-to-consumer brands like J. Crew realized this early, got out ahead of the curve, and are reaping the benefits. New companies selling other brands (e.g. Zappos) were notable as well. For the most part these companies built revenues through leveraging existing audiences (in the case of existing direct-to-consumer brands) or taking advantage of SEO and paid acquisition opportunities (in the case of wholesale retail). There is still some growth left to be had in this area, but there is one segment in particular that will drive e-commerce growth over the next few years: existing designer brands who previously only sold wholesale moving into direct-to-consumer online.

[Read more...]

The Problem With “Warby Parker for X”

Direct to consumer online-only retail is not disruptive because its online. Do not mistake good marketing for a new and innovative business model. Warby Parker was a special case driven by very interesting market dynamics that don’t apply to the other companies in the space. Let’s dive in and take a look.

Over the last few years there has been a surge in new direct-to-consumer online-only companies, particularly in the apparel space, companies like Everlane, Pickwick & Weller, Beckett Simonon, and Pistol Lake. These new models have been framed as [disruptive innovations] ( by investors, the media, and of course the startups themselves, but the fact is that they are not offering something that is fundamentally different from the market incumbents from a business model perspective. These verticalized retailers are mostly trying to replicate what Warby Parker has accomplished with eyewear, but unfortunately that category had very specific advantages that don’t apply to other vertical apparel segments in general. Warby Parker used online distribution to put the first chink in the armor of a huge monopoly, Luxottica.

[Read more...]

The Future of Analytics – The Data Platform

The future of analytics is event based, with events tracked back to individuals. People not pageviews. There are a couple obvious trends that are driving this:

  1. Tablet and mobile are now growing at a much faster rate then the web, as mobile moves to take over web in terms of market share we’ll see apps driving more usage than websites.

  2. Websites are moving from thin-client to thick-client and single page applications and other similar Javascript heavy sites are becoming the norm. Client-side frameworks like backbone.js, ember.js or new full-stack frameworks like Meteor are further enabling this.

[Read more...]

New Startup? Think Twice About Mobile First

Since Fred Wilson wrote the original post about “mobile first” companies nearly two years ago, there has been a large uptake in this philosophy. In general, this makes sense as internet usage is moving rapidly off the desktop and onto mobile devices. Every day more new services are being developed that are truly “Mobile Only” where their web presence will always be relatively minimal (Uber, Hotel Tonight, Instagram, etc.).

[Read more...]

Growth Hacking: A Primer

Introduction to Growth Hacking

Almost every large successful internet company right now has a team dedicated to growth, but it isn’t some sacred thing limited to companies with millions of users. Since there seems to be a lot of interest (… around what the process is like, hopefully this post can shed light on the situation.

So what is growth hacking? I think of it as the practice of gathering data, exploring that data, and exploiting knowledge uncovered from that data in a systematized way to directly further the business goals of your company. Its more than just a role fulfilled by one person or a team of people, its a way of thinking you can use to make sure you’re getting the most out of the work you are putting into your company. It works for small startups (we prioritize development using this framework at Lookcraft all the way up to huge companies (Facebook’s growth team has massive influence across the whole organization).

[Read more...]