bartleby, an online homework help service, was exploring opportunities to improve profitability and retention. We identified that customers were seeking faster response times and support for a wider range of math topics. After introducing a math solver product, we lowered our average answering costs per question and improved 30-day retention by 9% through faster response times and covering a wider range of math topic.
Background
bartleby is an online homework help service for college students studying STEM courses. It provides access to a library of step-by-step homework solutions, on-demand homework help with subject matter experts, and one-on-one tutoring.
At the time, the company was beginning to explore opportunities to improve profitability and retention through its math solving experience after identifying a majority of questions submitted by students were math equations and declining customer satisfaction scores.
Process Overview
Insights
Through customer research, data analysis, and market research, we discovered:
Expectations for faster response times: Students expected a much faster response time to help solve math equations compared to bartleby’s 1-hour average. The introduction of math solvers in the competitive landscape enabled competitors to generate step-by-step solutions to math problems in seconds.
Wide range of response times: bartleby’s math subject matter experts were not evenly distributed across all math topics leading to certain topics having faster response times and some having slower responses times than the 1-hour average.
Wider math coverage: Based on student feedback, they were looking for help on some math topics that were not fully covered by the current pool of subject matter experts.
Behind in the math solver space: From a strategic perspective, bartleby was behind the competition in providing a math solver product. A math solver allows a student to enter a math equation and then generates a step-by-step solution with explanations. Many direct and indirect competitors such as Chegg, Symbolab, and Photomath were providing fast math help through math solvers.
Hypothesis
We believed offering a math solver product would improve customer retention through faster responses times and better topic coverage in math along with improving profitability through lower answering costs per question.
Researching math solvers led us to believe that we could:
Improve response times: Drastically drop our response time to seconds from the current 1-hour average for math equations.
Improve math topic coverage: We could build a product that better met our customer’s math topic needs, or leverage partners that had better coverage.
Lower answering costs per question: After running financial models, we believed there was an opportunity to reduce our average answering cost per question through either building, partnering, or buying a math solver product.
Validating Demand
To assess demand for a math solver feature, we conducted a 3-week painted door test along with a survey.
The test displayed a new panel on the homescreen describing the benefits of a math solver with a call-to-action as if it was already built. Once a user clicked the button, we led customers to a survey to capture information about their math problems and topics they needed help with.
From the test, we learned that there was a large interest in the feature, and additionally, we learned more about math topics that we currently didn’t cover or had little support for.
Build vs. Buy vs. Partner
As we assessed our path to delivering a math solver product, we considered either building the product ourselves, partnering with an existing provider, or buying a math product solver.
The main considerations were:
Internal Expertise: Did we have the internal expertise to build this product?
Time to Market: How long would it take us to get to market?
Costs: What were the costs involved?
Transaction Risks: Are there any notable transaction risks or complexities?
Ultimately, we decided on partnering with a very reputable vendor known for its expertise in dynamic computation answering and leveraged their white-label solution for math solvers. Our overall deciding factors were:
Lack of internal expertise: We didn’t have the required internal expertise to build a math solver on our own and needed to either partner or buy a solution.
Time to market and transaction risks: We wanted to get to market as soon as possible to be competitive in the space and minimize customer churn. Integrating with an existing solution would let us get to market quickly and validate the solution while avoiding the complexity of buying a company.
Widest math coverage and new capabilities: With the vendor we chose, it provided the widest math coverage of the options and opened opportunities to expand into other computational services in chemistry and physics.
Simple and intuitive user experience: After running several user tests, we found that users found using our selected vendor the easiest to use and enjoyed using the OCR capabilities.
MVP
For the MVP, we decided to first launch our math solver on our new mobile app. We wanted customers to feel the simplicity of answering their questions and reach that a-ha moment by leveraging the snap-and-solve capability. This allowed customers to easily take a photo of their questions rather than typing them in and then having the question solved automatically.
The MVP consisted of:
Integrating with our third-party vendor to send questions and receive solutions
Supporting image and manual input of math questions
Displaying step-by-step solutions for various types math questions
Updating the UI with bartleby’s branding
Workstreams
Some of the workstreams involved in the development of the MVP included:
Defining Success: Defined success metrics based on data provided by tests and current engagement and retention metrics.
Product Strategy & Feature Definition: Prioritized feature alongside other initiatives, defined core feature requirements,and collaborated with design and engineering to develop designs, implementation plans, and usability studies.
Product Development: Managed the backlog, ensured efficient team operations by tackling roadblocks, and conducted user testing.
Product Launch: Coordinated with engineering a phased rollout and worked with marketing to define the marketing communication strategy of the new feature.
Result
Our hypothesis was validated, we delivered a math solver that improved customer retention and improved our profitability per math question. Our math solver successfully reduced response times and expanded our math topic coverage which lead to more math question submissions and improved our 30-day retention by 9%.
