Codecademy, for those unfamiliar, is a B2C tech education platform that focuses on taking total beginners and getting them to a medium level of technical skills.
The company started in 2011 and built what I still think is the best free learn-to-code product online.
As all “advice” really depends on the context that it’s coming from, here is where we were at as a company in roughly 2017 :
- Very strong, low-cost free user acquisition driven by the free product, excellent brand name, and strong SEO ranking
- Product market fit in the free product for users who are early-stage learners (i.e., not professional software engineers)
- Short/Intermittent usage patterns across all users — E.g., they show up for a few weeks/months to learn something, then leave for a while, then maybe come back in a few months/years. Not the best use case, but a solid one
- A paid product that was built but was still being figured out.
TL;DR — We had a lot of free users but were terrible at converting them to paid and monetizing them.
The final bit of context is that I am a product manager, so most of our work was on the product rather than in marketing channels. We did have a great marketing team; however, I’ll mainly talk about what I implemented firsthand and saw work.
Like most first-time growth teams, we made many classic mistakes in our first year.
The most painful/memorable of those are:
1. Not understanding what to A/B test vs. just shipping it
When we first tried to set up an A/B testing program, we got way too lost in the concept of “measuring” all of the impacts that we were having and not making any “assumptions”. That was dumb.
We debated if we should have a pricing page on the site and how we might measure that impact. I remember that adding a pricing page to the header had a small to medium size impact on conversion, which is great in theory.
However, I would consider that one a loss, as we should have had this in the header the whole time.
2. Launching giant projects without enough due diligence
The flip side is that you should not assume that A/B testing is an excuse not to do the due diligence you need on big projects. We shipped 3 different pricing page “layouts” to see which was more effective.
I think it took 2 months to run, was super expensive from a design and engineering perspective, blocked other tests, and we didn’t learn anything in the end.
3. Thinking of ideas that were “fun” or “fancy”
One of the tenants of “Growth” is that you should be able to test any idea you have a solid hypothesis behind. While I think this is valuable, it’s way more important to start with a solid foundation of best practices in your area.
The “fancy” ideas (such as “running subway ads” or “partnering with influencer”) only work if you have a solid funnel to drive this traffic into.
4. Working in the wrong areas of the application
As I mentioned above, we re-designed our pricing page twice before realizing that a very small % of users purchase right from that page. Most are purchased right from the product itself without going back to pricing.
5. Not iterating on good ideas that didn’t work out the first time
In retrospect, we moved on too quickly from ideas we had decent confidence in that didn’t work the first time.
If you are hitting a real customer problem, you did your research, and V1 isn’t working out, then I would take another shot.
6. Not implementing best practices fast enough
As mentioned above, if you don’t A/B test the basics, just ship them.
Combining this with the friction of getting all the necessary tooling set up, learning how to run an A/B testing program, and building out the team…. this wasn’t our best year.
Additionally, B2C SaaS and subscription businesses can be tricky to grow for a few reasons, especially when paired with a free product/tier.
- These businesses can take a lot longer to grow because you aren’t going to close a big enterprise contract, and scaling paid media is more challenging.
- The value you capture from users is spread across their lifecycle, so being confident about your LTV and the spending in marketing channels is hard (not impossible, but hard)
- Any time you do something like test new pricing (which you should), you can’t really know the impact on LTV because those new users haven’t reached the end of their lifecycle.
- When you’re also running a big free product, it is hard to run direct response style paid media effectively as you can’t really prevent users from finding the free product.
- Understanding payback periods for ad spend gets more challenging when they might spend variable amounts of time in the free tier based on the audience.
Recurring revenue businesses are basically structured like this diagram.
Many companies go wrong because they only focus on making acquisition cheaper and faster when you should have been systematically improving LTV to afford higher acquisition costs.
There is a Dan Kennedy quote, which is something like:
“The business that can pay the most to acquire the customer will win.”
Assuming that you don’t have a bunch of VC money light on fire (and even if you do), the best chance that you have in the long term is figuring out to better monetize each monthly cohort of users and increase your overall LTV, which will allow you to spend more to acquire more customers profitably.
Here is what I would advise companies in a similar place to do. This is a path toward reliably making improvements, not seeing giant lifts.
1. Figure out monetization early in your product/company’s life
Everything gets more expensive to change the longer you wait. It involves convincing people, changing more code, retraining more sales/customer service people, etc.
Every cohort you under-monetize during your early years is money you never get back.
2. While speed is essential in all businesses, it’s extra important in subscription businesses
Subscription businesses are slower to scale than most B2B businesses because most of your customers will pay incrementally across their lifecycle, and you are not likely to close a big enterprise account that allows you to collect a lot of cash upfront.
Because of this, every missed cohort of users you didn’t monetize well is lost forever. The faster you get the best practices running, the more revenue you’ll have to compound in the following years. Don’t overthink the fundamentals.
3. Work from the bottom of the funnel upwards unless you have a good reason not to
You’re active paying users are hands down the most valuable to you. You don’t want to lose them for things like payment failures that could have been prevented.
If you are in a company that drives decisions based on A/B tests, you will be beholden to a concept called minimum detectable effect (MDE).
- MDE is the smallest lift you can measure with statistical significance (which basically translates to confidence).
- So if you have a high MDE, say 20%, and you ship a test that improves something by 19.99%, you won’t be able to measure with statistical significance. There are ways of compensating for this, but this makes testing hard.
- The worse your overall conversion is, the larger the MDE will be, which makes A/B testing harder.
- But the upside is that improving things at the bottom of your funnel will lower (a good thing) the MDE, which raises the accuracy of your ability to measure things above it in the funnel.
4. It’s hard to convince people to pay you money, so start by finding all the people who are trying to pay you, and it’s not working
Find all the dead ends and confusion across your product and fix those.
Ensure the checkout page error states are clear, that any payment integrations aren’t broken in specific browsers, etc. Are the bugs that happen during certain edge cases in important parts of the app?
5. Try to keep complexity low
Your goal is to keep development and operational speed fast, so check that any changes you make are worth the maintenance costs.
- For example, Suppose you find that setting up custom price structures per country raises revenue.
- From now on, the finance team will need to build multiple models to forecast revenue, the customer service needs multiple sets of auto-responses, etc.
6. Focus where most users are and focus on improving clarity
Try to find the places in your product that most users are converting from and ensure that you’re setting context correctly the step before.
One of our most successful tests on our homepage changed the primary CTA from “Start Coding” to “Sign Up.” Most of the time, we were cute with copy or messaging; it hurt us.