How to Lose Money on Paid Marketing

Paid marketing is a powerful tool, it’s one of the few ways to scale user growth but if not measured correctly it’s also one of the best ways to drain your bank account, fast. If you’re going to use paid marketing as a sustainable growth mechanism you have to know a lot more than if it’s driving new visits, you have to know if its profitable. It seems like something that is simple enough to calculate, but most marketers actually get it wrong. Let’s explore why.

How To Calculate Profitability

Most marketers have settled on a simple metric for measuring how profitable their channels are called return on ad spend (ROAS). ROAS is defined as the revenue a user generates divided by the advertising costs. It is usually calculated in two ways, across all channels together and for each channel individually. To understand the performance of our entire marketing budget for executive level reporting we would calculate it across all channels, but to make marketing decisions about what channels are working and which ones are not we would calculate it for each channel individually and compare the results. In equation form it can be stated like this:

ROAS = (Revenue)/(Marketing Spend)

Back to Algebra Class
This equation looks simple but is actually tricky because Revenue and Marketing Spend use different units. If you think back to your Algebra classes I’m sure you’ll remember having to solve problems that combined units like miles/minute and miles/hour into a single equation. In order to get the right answer, you first needed to change miles/minute to miles/hour. The ROAS equation is similar. One unit is Cost-Per-Visit (Marketing Spend) and another is Revenue-Per-Conversion (Revenue). In order to get the right answer we need to convert revenue-per-conversion to revenue-per-visit.

Calculating Revenue Per Visit
The Marketing Spend part of our equation above is easy to solve since we already know our cost-per-visit but Revenue will require some work. We know our revenue-per-conversion, but to turn that into revenue-per-visit we need to give a value to each visit based on the conversions it contributed to. Visits that never led to a conversion are easy to value, they get a value of $0. The difficult visits to value are those that did lead to a conversion. In order to give these visits values we need to come up with a set of rules that determines how to to value them. We call this set of rules an Attribution Model, it’s the key to calculating ROAS.

Attribution Models

Attribution models are a set of rules that determine how we convert Revenue-Per-Conversion into Revenue-Per-Visit so we can fairly compare our revenues to our marketing costs. There are two basic types of attribution models.

Single-Touch – The Old Way
A single-touch attribution model is one that gives credit to only one visit for a conversion. This is the most common way marketers today attribute revenue. It is easy to calculate but not very accurate.

Some common single-touch models are:
– First Click (give credit to the first visit)
– Last Click (give credit to the last visit before conversion)

Multi-Touch – The New Way
A multi-touch attribution model is one that gives credit for a conversion to a number of visits rather than just the first or last visit. This is the new way that smart marketers are using to attribute revenue. It is harder to calculate than single-touch attribution but is far more accurate.

Some common multi-touch models are:
– Uniform (give credit evenly to all visits)
– Parabolic (give credit to all visits but more to the first and last visits)
– Time Decay (give credit to all visits with more credit going to later visits)

Attribution – An Example

In order to understand why multi-touch is the “new way” lets consider a specific example. We’ll take a last-touch attribution model and a multi-touch attribution model and compare the differences in ROAS and channel performance.

Assume we have 5 users who visit our site for the first time on May 1st or May 2nd, these users then each make a $200 purchase on May 9th with a number of other visits in between their first visit and purchase. The user visits look something like this, with each visit channel color-coded for convenience (Facebook is blue, AdWords is green, Retargeting is purple):

Screenshot 2015-05-29 16.20.28

We also have some advertising across different channels over each of those days:

Screenshot 2015-05-29 16.22.23

Single-Touch Attribution – Last-Click Model
Now lets check out what last-touch attribution looks like in this scenario. We’ll ignore dollar values for now and say that each conversion has a value of 1 to make this simple. Let’s see how our different marketing channels are credited here. (Facebook is blue, AdWords is green, Retargeting is purple). If we add up the credit for each channel it looks something like this:

Screenshot 2015-05-29 16.34.13

As you can see, the channel that each user visited last gets 100% of the credit for the conversion. In this case Facebook was the last channel two times, AdWords was the last channel two times, and Retargeting was the last channel once.

Now let’s include the spend and see what it looks like. Remember each conversion is worth $200. The ‘reporting window’ just means we’re only looking at what is happening during those dates. For example, when in Google Analytics you select to look at the last 7 days, those 7 days are your ‘reporting window’ even though there may be other user activity happening outside that timeframe. Since we only have the 5th to the 9th selected as our reporting window we’ll add up all the marketing spend and all the attributed conversions in that time period. The result looks something like this:

Screenshot 2015-05-29 17.30.42

This looks great! From the 5th to the 9th we spent $600 on marketing and made $1000 in revenue on the users acquired from those marketing efforts for an ROI of 1.67x our marketing spend. Based on our last-touch attribution model for this sample of users it appears that AdWords and Facebook are equally good at driving conversions, and each are twice as good at
driving conversions as retargeting is on a per-dollar basis. Before we go tell our boss we need 10x more marketing budget and start allocating more spend to Google and Facebook than Retargeting lets give multi-touch attribution a shot.

Multi-Touch Attribution – Uniform Model
As we did before with last-touch attribution we’ll ignore dollar values for now and say that each conversion has a value of 1. Lets see how our different marketing channels are credited here. (Facebook is blue, AdWords is green, Retargeting is purple). In this case, instead of giving value just to the last click we’ll split it up evenly between every visit that drove a conversion. If we have one conversion for a user and 4 visits as we do in this case, each visit will get 1/4 of the credit for the conversion. The way it works out looks like this:

Screenshot 2015-05-29 17.27.22

Now lets include the spend again and see what it looks like. Remember each conversion is worth $200, so we just multiply the partial credit amount (0.25 for all visits in this case) for each visit by the conversion value for each conversion ($200 for all users in this case). The result looks something like this:

Screenshot 2015-05-29 17.32.26

That is quite a bit different than our last result, not only is the overall ROI worse 0.83x vs 1.67x but the per-channel numbers actually tell a completely different story. Here’s how last-touch and uniform attribution compare overall:

Screenshot 2015-06-16 16.09.12

Our more robust multi-touch model tells us we actually have 50% of the ad performance we thought we did, crazy! Here is how the it looks if we compare by marketing channel:

Screenshot 2015-05-30 14.59.26

Our single-touch model would suggest we move budget away from Retargeting and towards Facebook and AdWords since they have almost twice the ROI (2x vs 1x and 1x respectively), but our multi-touch model actually tells a different story with Retargeting performing 33% better than Facebook and AdWords (1x vs 0.75x and 0.75x respectively). It also suggests that we shouldn’t be expanding our budget at all yet, at least not without more in depth analysis on user lifetime values since we’re still losing money on our advertising.

The takeaway here? The attribution model we select can have large effects on how our spend is performing, using a naive model (or no model at all) can leave us in the dark about how our ad spend is performing, leading us to think we are making money while we’re actually losing it.

Implementing Attribution

Prescribing which specific attribution model is the correct one for your company is out of the scope of this post, but any model that gives credit to each visit along the path to conversion (multi-touch) is going to be much more accurate than one that only gives credit to a single visit (single-touch). Moving from a single-touch attribution model to multi-touch one can have big impacts on spend allocation decisions and can significantly improve advertising ROAS. As we saw in the example above.

It’s a lot of work to build out accurate multi-touch attribution, but there are third-party solutions out there that can simplify the process. My company Interstate makes a pretty good one in my opinion! Regardless of whether you build or buy, if you’re going to use paid marketing as a sustainable growth mechanism you absolutely must have a clear understanding ROAS. In order to that you must move past single-touch attribution, without it you might as well be guessing.

Metrics Debt

In the same way under-investment in technical rigor can result in long-lasting negative effects on a codebase when left unchecked, under-investment in metrics can lead to long-lasting negative effects in your product organization.

At Quint Growth we work with companies every day to improve how they measure, understand, and act on data. We review many metrics setups and talk to companies about how they approach the problem of understanding and optimizing growth. We find that key questions that companies need answers to in order to make important product and business decisions are often either poorly understood or require quite a significant amount work to answer. This doesn’t vary by company stage, we’ve seen this just as much in small companies as we have in large successful ones. I’ve never spoken to a single company who has complained about their over-investment in metrics, its chronically under-invested in by all post-product-market-fit companies.

There are a number of effects that under-investment in metrics causes:

[Read more…]

Retention Is King

This was originally posted as a guest post on Andrew Chen’s blog here.

There are too many companies asking, “How do we acquire more users?” that should instead be asking “How do we get better at keeping the users we already have?”.

Its easy when approaching the problem of growth to think that you just need to get more users, after all that seems to be the very definition of growth. However, if you take a step back though and think about growth as the maximization of user-weeks over time, it quickly becomes apparent that focusing on retention has a much larger effect than topline growth. This is also much more of a sustainable growth mindset. Rapid user growth followed by rapid user attrition is an indicator of unsustainable growth. Strong retention of users over time is a good indicator of product-market fit, something you’re hopefully looking to achieve anyway.

Viral Factor and Retention

At a high level, retention is more important than virality because if your users don’t stick around they are not able to invite others to your product over an extended period of time. If you have high retention and no virality you will sustainably grow your user-base over time. If you have high virality and no retention you will not. Between these two extremes it gets a bit more complicated. In order to explain in detail we first need to review a couple of terms: viral factor and retention.



It will help as you follow this post to use our in-house growth model to play with the numbers yourself, the graphs we reference later on in this post are derived from it.

Viral Factor

This describes the growth rate of a site or app based on invitations from existing users of the service. This is often called k factor.


i = number of invites sent by each customer

c = conversion rate of those invites (#signups/#invites)

k = i * c

Viral factor on a weekly basis usually looks something like the graph below. This varies for different products, but I’ve seen this shape again and again across the products we consult on. It is front-loaded like this for three reasons:

  1. The effectiveness of onboarding invitation flows.
    Onboarding is one of the few times you have a high level of user attention towards completing a specific goal (signup) and when you instruct users to invite others they often will without thinking much about it.

  2. Users’ level of excitement.
    Humans are most excited by new things, this applies to internet products as well. The excitement users get when trying a new product leads them to share more, but this sharing tapers off as product becomes normal in their everyday lives.

  3. Low invite saturation of users networks.
    When a user first starts using a product they know many more people that don’t use the product than do use it. Over time they share your product with others they know. Eventually, even a user who is very passionate about your product has nobody left to share it with that hasn’t already heard about it leading to lower virality as time goes on. This can also be an issue if your company grows very large, but that is a good problem to have.

Screenshot 2014-05-15 16.55.26

Retention

This is the number of users that stick around from one time period to another. There are two ways to express retention, overall retention and week-to-week retention:

  1. Overall Retention

    Overall retention is cumulative over time. If you have 30% overall retention in Week 3 it means that 30% of your users who started at the beginning of Week 1 are still around in Week 3. This is how companies normally express retention when discussing it internally.

  2. Week-to-Week Retention

    For growth purposes its often useful to look at retention on a week-to-week basis instead. Week-to-week retention is how many users move from one week to the next. If Week 2 has 40% overall retention and Week 3 has 30% overall retention than our week-to-week retention from Week 2 to Week 3 is 75%. If week-to-week retention is below 100% it means we’re still losing users.

Retention on a week-to-week basis usually looks something like the curve below. It is lowest from the first week to the second week and approaches 100% as time goes on.

Screenshot 2014-05-15 16.55.20

Why is retention so important?

In order for virality to be more important than retention your viral factor must be greater than your overall retention up to that point in time. We’ll prove this mathematically later in this post. The math is hard to simplify exactly, but there is a basic rule you can follow which approximates it. If you take only one thing away from this post it should be this:



Do not focus on improving virality unless your overall retention is stable, not continuing decrease after some reasonable period of time.



To help illustrate this, lets look at a few examples:

  1. Your product has a very high immediate viral factor.

    If your product is front-loaded with invites that are sent out and accepted at a high enough rate, it is possible to achieve an immediate k > 1 viral factor. In this case, if your invites have an acceptance rate that is quick enough, your monthly active user numbers will continue to grow even if you have zero retention past the first-use of your app/site. Growth of the high virality, low retention type is almost always unsustainable as the viral loops being exploited to attain quick k > 1 virality eventually expire. This has been the downfall of many companies that appeared to grow fast, raised a lot of money on that growth, then quickly died. Like Viddy…

    viddy


  2. Viddy growth model with viral channels working looks like this.

    Viddy growth model with viral channels broken looks like this.

  3. Your overall retention is high and decreases slowly as time goes on, but you have strong virality.

    In this case, products actually do see long term benefit from focusing on increased virality, but it’s often a false signal. Its only worth focusing on virality if you think you can improve virality more than overall retention. If you switch focus too early it will lead to sub-par growth metrics. This is because the compounding effects of retention improvement are much stronger than those of virality improvement.



    We can illustrate this easily with our retention/virality simulator and setting week-to-week retention and virality to be equal. We can then measure the effect of proportional changes to one of the other in terms of number of users that we have at some point in the future. In the real world virality is not likely to ever be equal to week-to-week retention, but for purposes of this example please disregard that as it helps illustrate our point in the clearest way possible.



    Base Case – Equal Virality and Retention: ~88k users total in Week 7, 44k from retention and 44k from virality as seen in the stacked bar chart.



    20% Increased Week 1 Virality: ~110k users total in Week 7, 53k from retention and 57k from virality.



    20% Increased Week 1 Retention: ~125k users total in Week 7, 65k from retention and 60k from virality.



    As you can see here, changes in retention have long term effects that have a greater effect than equivalent changes in virality.


  4. Your overall retention is high and stable.

    If you have maximized your retention to the point where you think you can increase virality more than overall retention, even accounting for the compounding effects of retention, then it makes sense to focus on virality.

Prove It!

It’s easy to model growth in Excel and there are some great models that have been shared online to help do this. The one I use is available for download at http://bit.ly/growthmodel. It’s a slightly modified version of this great one from Rahul at Rapportive. It will give you a nice overview of how you can expect to grow if you plug in some numbers. From a user accounting perspective this is great, but its a bit hard to conceptualize how growth actually works by looking at it that way. To get a bit of a different perspective, building a tree to see exactly where users that exist in a given week come from is quite helpful.

In the tree below w0 represents some set of users that start at time 0. Each level of the tree is a week in time. At each subsequent level you get users that stick around from retention or are invited via user virality. The number of users at a given node is just the product of all the nodes leading to that point. The coefficients represent the viral and retention factors over time, v2 is the viral coefficient in week 2 from the above virality graph, r3 is the retention coefficient in week 3 from the above retention graph, etc.

Screenshot 2014-05-13 17.24.01

The number of users at any given level can be simplified into a recursive equation.

Screenshot 2014-05-13 17.15.28

As you can see, this matches exactly what we saw in the tree graph above. The leading viral factor (current virality vi) matters relative to overall retention (trailing product of rn‘s).

Visualizing Retention and Virality

For the client work that we do at Quint Growth we built a tool to plug in numbers and visualize the retention/virality tree. Its the same tool we mentioned in the beginning of the post and used in Viddy example above. We’ve found it incredibly useful for visualizing how much more important retention is than virality (and the few cases in which it is not). We’ve made it available at http://quintgrowth.com/growthmodel.html and hope you find it as useful as we do!



Special thanks to Isaac Hodes for help with the d3 visualizations for the retention/virality visualizer.

Add Funnel Steps to Improve Conversion

In my last post I discussed how an understanding of consumer psychology can be extremely helpful for understanding how to design funnels that convert. In this post I’m going to give a concrete example of how I did this at Lookcraft to drive 18% conversion through a funnel with 18 discrete questions and 4 major steps. I got 18% of users who hit our homepage to enter their email and password, information about their fashion taste, and full sizing information. I’ve used these same tactics across multiple different products to similar effect with other companies I’ve worked with or consulted for and hope they will be useful for you as well.

[Read more…]

A/B Testing is Expensive

Over the last few years the startup community has really gotten behind A/B testing and hyped it up quite a bit. There is a more nuanced point about the downsides of A/B testing that needs to be understood: A/B tests are very very expensive for most startups at the time when they matter most, early in their formation.

[Read more…]

Goodbye Wholesale Brands

The next few years of e-commerce growth are going to fundamentally change the way soft-goods retail works today.

forrester2.png

E-commerce has been growing rapidly for the last 6 years with equally rapid projected future growth. To date the growth has been mostly powered by existing brands moving rapidly online. Great direct-to-consumer brands like J. Crew realized this early, got out ahead of the curve, and are reaping the benefits. New companies selling other brands (e.g. Zappos) were notable as well. For the most part these companies built revenues through leveraging existing audiences (in the case of existing direct-to-consumer brands) or taking advantage of SEO and paid acquisition opportunities (in the case of wholesale retail). There is still some growth left to be had in this area, but there is one segment in particular that will drive e-commerce growth over the next few years: existing designer brands who previously only sold wholesale moving into direct-to-consumer online.

[Read more…]

The Problem With “Warby Parker for X”

Direct to consumer online-only retail is not disruptive because its online. Do not mistake good marketing for a new and innovative business model. Warby Parker was a special case driven by very interesting market dynamics that don’t apply to the other companies in the space. Let’s dive in and take a look.

Over the last few years there has been a surge in new direct-to-consumer online-only companies, particularly in the apparel space, companies like Everlane, Pickwick & Weller, Beckett Simonon, and Pistol Lake. These new models have been framed as [disruptive innovations] (http://en.wikipedia.org/wiki/Disruptive_innovation) by investors, the media, and of course the startups themselves, but the fact is that they are not offering something that is fundamentally different from the market incumbents from a business model perspective. These verticalized retailers are mostly trying to replicate what Warby Parker has accomplished with eyewear, but unfortunately that category had very specific advantages that don’t apply to other vertical apparel segments in general. Warby Parker used online distribution to put the first chink in the armor of a huge monopoly, Luxottica.

[Read more…]

The Future of Analytics – The Data Platform

The future of analytics is event based, with events tracked back to individuals. People not pageviews. There are a couple obvious trends that are driving this:

  1. Tablet and mobile are now growing at a much faster rate then the web, as mobile moves to take over web in terms of market share we’ll see apps driving more usage than websites.

  2. Websites are moving from thin-client to thick-client and single page applications and other similar Javascript heavy sites are becoming the norm. Client-side frameworks like backbone.js, ember.js or new full-stack frameworks like Meteor are further enabling this.

[Read more…]

New Startup? Think Twice About Mobile First

Since Fred Wilson wrote the original post about “mobile first” companies nearly two years ago, there has been a large uptake in this philosophy. In general, this makes sense as internet usage is moving rapidly off the desktop and onto mobile devices. Every day more new services are being developed that are truly “Mobile Only” where their web presence will always be relatively minimal (Uber, Hotel Tonight, Instagram, etc.).

[Read more…]

Growth Hacking: A Primer

Introduction to Growth Hacking

Almost every large successful internet company right now has a team dedicated to growth, but it isn’t some sacred thing limited to companies with millions of users. Since there seems to be a lot of interest (http://andrewchenblog.com/2012/05/11/how-do-i-learn-to-be-a-growth-hacker-wor… around what the process is like, hopefully this post can shed light on the situation.

So what is growth hacking? I think of it as the practice of gathering data, exploring that data, and exploiting knowledge uncovered from that data in a systematized way to directly further the business goals of your company. Its more than just a role fulfilled by one person or a team of people, its a way of thinking you can use to make sure you’re getting the most out of the work you are putting into your company. It works for small startups (we prioritize development using this framework at Lookcraft all the way up to huge companies (Facebook’s growth team has massive influence across the whole organization).

[Read more…]