bartleby | Free Trial Experiment

Background

bartleby recently launched and was focused on increasing its number of subscribers.

The two main product offerings, at the time, were:

• Ask a Question (Q&A): subscribers could ask a subject matter expert a question and get an answer in 1-2 hours.
• Solutions Library: access to a library of textbook solutions and previously answered Q&A questions. 

The team also recently discovered that publishing Q&A pages publically was an effective way to drive organic traffic through search. As a result, they were also looking for ways to get more students to ask questions. 

Insights

We analyzed our funnel and noticed a significant number of users would get to the payment screen but then abandon the flow; they were very motivated until the last step.

After conducting interviews and surveying them, I found:

• Abandoned users were very budget-conscious and were not confident that the Q&A service could be as good as it was marketed.
• While subscribers greatly valued the Q&A service and didn’t use the Solutions Library as often because they needed help with specific questions that weren’t in the library. The analytics confirmed this and I found that subscribers used most of their monthly questions. 

Hypothesis & Experiment

I hypothesized that if we gave abandoned users free questions to ask, they would ask a question and experience the value of the product, and subsequently become a subscriber to use more of the product. As a secondary goal, I wanted to see if this was a viable way to get students to ask more questions, so bartleby could  create new Q&A pages. 

What we did:‍

• Added two free questions to 500 user accounts that abandoned in the prior two weeks.
• Emailed them about the offer.
• Monitored email open rates, questions asked, and the conversion rate to subscribers.

Results & Learnings

Approximately 30% asked at least one question and ~6% of the test users subscribed. The free trial experiment was a success and proved to be incremental in recovering abandoned users. The ~6% recovery rate represented 2x the current recovery rate of existing tactics for the product. We therefore learned that offering free questions was a valuable lever to bring back abandoning users and get some to subscribe and grow the library. 

Takeaway

This was the first experiment leveraging a free trial to improve the recovery of abandoning users and, ultimately, the conversion rate at bartleby. After we productionalized the free trial experience, we continued to test different variations which led to increasing the user base by 32% and the Q&A library by over 83x, +1.9MM questions in 18 months.