Rating Decline of Leading Taxi Aggregator| Root Cause Analysis
Problem Statement — You are the PM of the leading Taxi aggregator. The Appstore/ Playstore rating dropped from 4.5 to 3.8 in the last 2 weeks. Find the root cause of the problem and come up with solutions to solve the problem.
Understanding the problem
To understand the given problem statement better, we should address the following questions:
- Is there any specific geography that is contributing to the drop in ratings?
- Is there any specific demographic age responsible for the drop in ratings?
- Have we seen a similar kind of drop in the past or any kind of pattern?
- Are the bad ratings received from the new users or existing users?
— It’s uniform
Since none of the above factors are affecting the drop, we can begin analyzing the problem.
Internal and External Factors
Before we get into the user journey, analyzing possible internal and external factors affecting the ratings will give us a good understanding of possible areas to analyze in the user journey.
- A new version of the app was launched.
— Yes, there was a recent update to the app about two weeks ago on both iOS and Android platforms.
- Changes in the product.
— In the recent version, we made changes to the algorithm of allocating the ride services.
- The number of rides going down in the past two weeks.
— Yes, the number of rides has gone down by approximately 15%.
- Changes in UI/UX in the recent update.
— Yes, there were UI changes made on the payment method page.
- Changes to marketing strategy.
— No Strategic changes were made in the last two weeks.
- Changes in policy by the government
— The government didn’t change policies regarding mobility services but there has been a steady increase in the number of covid-19 cases across the nation.
- A product that is changing users’ behavior
— Since the start of the pandemic, users are not comfortable traveling with a stranger and they are moving towards products that provide subscription-based services at a competitive price.
- Strategy changes by competitors
— No changes to the product or the way of marketing by the competitors. They have been doing it previously and it has not affected the ratings of our app.
- Bad PR like some incident of misbehaving by the driver or drivers getting covid-19 positive which got picked by the media.
— There have been some instances in the past of bad PR but none of them were in the last two weeks
- Metric changes to the way Appstore/Playstore rates an app
— No Changes are made to the way apps are being rated as no significant drop in rating has been witnessed with other apps.
- Have there been any reports of data leaks of users’ information that are being tracked by the app?
— No, there have been no such incidents.
Now let’s go through the entire funnel to understand the user journey and the problems faced.
- Can we confirm with the analytics, if there has been a drop in user engagement at any point in the user journey?
— Yes, there are two stages where we could see a drop in user engagement. First, select payment method and second is while allocating the driver.
Payment Method Selection
- What changes are made to the Payment methods page?
— Since the pandemic, people started opting for hands-free payment methods for safety reasons. Observing this change, we changed the default method of payment from Cash to UPI payment.
- What percent of users did drop at this stage of the journey?
— When the new version was released, which was two weeks ago, the drop was about 3% at this stage but if we look at the data of last week the engagement has improved and the drop is just 1%.
The change in default method cannot be the reason for the ratings to drop and can be neglected as:
- The changes were made based on data received and were applied to improve the user experience.
- Only 3% of the users seemed to have a bad experience, this change was made based on the response of the majority of the users as they preferred to make payments using hands-free payment methods.
- Looking at the drop, we can say that a small number of users still preferred to use the traditional method of payment and were not able to find a way to change the default method.
- The increase in user engagement over the past week suggests that the small group of users can change the default method or started using the hands-free payment method. This can be further clarified if we look into the analytics.
- What percent of users drop while waiting for a ride to get allocated?
— Based on the data received approximately 15% of the users drop from the platform while waiting for the allocation.
- What was the average time of allocation before the update and the current average time?
— Previously, the average time of allocation was 30 seconds and after the update, it has increased to 120 seconds.
This could be the reason not only for the drop in engagement but also for the drop in ratings.
Analyzing External Factors
Before we conclude, we should also look at some possible external reasons for the drop in ratings.
Increasing Covid-19 Cases
An increase in the number of covid-19 patients across the country can affect the number of daily active users (DAUs) or Monthly active users (MAUs) but won’t result in bad ratings.
This won’t affect ratings as the users are not using the app and they won’t have a bad user experience.
Products with subscription-based models can alter user behavior. Users think to switch to the new product because of the attractive price and safety regarding Covid-19. 25% of the users after experiencing or knowing their services, feel our product is expensive. This could lead to a bad user experience and eventually to a bad rating.
Root cause for bad ratings
Based on the hypothesis, it can be concluded that both internal and external factors collectively contributed to the drop in ratings.
- Due to the change in the algorithm of allocating the ride, the average time taken for allocation increased by 90 seconds per ride. This led to a bad customer experience and disappointed customers went on to give us low ratings
- A new product that is providing a vehicle on subscription has influenced user behavior. Users find the new product to be more safe and financially efficient than our product. Users complaining about high fare prices have increased significantly in the past two weeks.
Solutions and Impact:
- Solution- Check the time required to improve the algorithm of allocation. If the required time is high then switching back to the old algorithm, until the algorithm is improved, will be ideal.
Impact- The goal is to improve the customer experience by reducing the time of allocation. Switching back to the old algorithm will mean that the time taken for allocation is less than the current scenario. This will solve the problem for users for a short period and improve the user experience. With improved user experience we can see a rise in the ratings.
- Solution- Before launching any changes to the product or algorithm, release it for a small group of trusted users or within the team. The release of the changes then can be decided based on the feedback from users.
Impact- This will help us understand the adaption of algorithms in real-time and customer feedback can help us improve. Launching it to a small group will also not affect our ratings.
- Solution- Improve marketing campaigns around the product and safety measures taken regarding Covid-19 without mentioning the other products and make the users realize that we are a requirement-based product and not a subscription where people can get trapped by the reasonable pricing and pay unnecessary money.
Impact- Looking at a competitive price, users may choose the new product but this campaign will help them understand that they choose the product looking at the pricing but they may not require the vehicle for a longer period. This is why our product is more efficient as we provide services only when required. If users understand that paying more for a single ride when required is efficient than paying for a subscription where per ride cost will be low only if the user requires to use the vehicle daily then ratings regarding fares will be improved.
Disclaimer: All the scenarios and solutions discussed in this case study are hypothetical.