A Blockbuster OTT App Release

About the Client

With a user base of 5M+, streaming media & entertainment company is one of the fastest-growing subscription-based video-on-demand and over the top (OTT) streaming service offering in Telugu. Founded by one of the most influential companies amongst the Telugu movies production and distribution business houses, the company is all set to revolutionize Telugu content consumption.

Challenge

The latest version of OTT app, launched in Q1 2020, was facing major UX roadblocks. The app rating was taking a hit where it had downgraded to 2.7 and the app users were raising more than a hundred issues daily. This was the situation in October 2020 and to add to their business difficulties, there was a big launch event planned with the Telugu film industry and introduction of exclusive movie titles planned for the app in mid-November. Consumers of OTT apps are a diverse set – which varies with the type of handset, the type of network connectivity and the type of OS. Recreating real-time scenarios across the varied staging environments is very crucial to predict the user experience accurately. Added to this, repeated issues were marring the customer experience such as ANR (Application not Responding), Crashes and App getting hung, despite two Quuality Assurance (QA) teams working full time within OTT app premises. Also, typically most firms rely on cloud device farms and test automation to maximize the coverage for compatibility and app performance. These solutions come with its set of limitations such as:

Automated testing on device farms covers standard scenarios and not the edge cases – it is difficult to test issues like app response when a user is fast-forwarding the video, or performance on full-screen versus split screen.
Network bandwidth related scenarios that cannot be reproduced
Coverage is on fixed devices, upgraded OS and device models get missed out
If tested manually, multiple testers are required to cover more devices

Solution

Given the extremely time crunched business deadlines, crowd sourced testing was the way out! Crowdsourced testing for an app involves distributing the app to a group of users who understand the application and who are in turn headed by experienced app testing experts. The team at Oprimes, India’s largest digital assurance & crowd testing platform, put together a team of 50 test-users within 24 hours. A team of senior test managers helped prioritize and guide these users. But that was not all! A typical crowd testing cycle is planned during UAT (User Acceptance Testing) or Beta phase so that the functional sprint issues are all fixed. However, the need of the hour was to fix all the compatibility and performance issues within days, and hence Oprimes had to integrate crowd testing within the sprint cycle. This meant running daily test runs for each new build with 50 test-users and two internal specialists, identifying and re-testing hundreds of compatibility and performance-related issues on different Android, iOS and TV devices on a daily basis.

Key Highlights

Multi-device coverage: 80% of the target market was covered.
Speed: The team and processes were set up within 24 hours.
Efficiency: The first round of testing captured substantial number of issues within a span of 8 hours.
Corner Cases covered: The issues identified were across devices and included scenarios that cannot be produced in the test labs
Execution Excellence: An agile & collaborative process was followed between testers and developers to identify and fix the problems
Rigorous iterations: The process was repeated for 8 different builds within 12 days’ time.
Localization: OTT App supports content in two regional languages and testing across both of them required quality assurances

Services Used

Impact

A successful app launch within 3 weeks that delighted customers with a flawless entertainment experience. The play store rating soared up to 4.4 while the production ticket count went down drastically by 80%. This was the most stable build that OTT app had released, as per their firebase analytics where the crash ratio as well as the adaptation of that build among the end users was high.

Summary

OTT app released the most stable app version, on time, winning back customer validation which reflected in high app rating. This case study is not just relevant for an OTT app but to any kind of media, internet, eCommerce, telecom, entertainment app involving large volume of users accessing the app on diverse device models and networks. Such apps need to work effectively for millions of users on diverse user channels, and can easily lead to UX, compatibility and performance related issues due to limited multi-device testing done in the labs. Oprimes’ efforts in such a short span of time made a huge impact to the client and this was well recognized as well. The client leadership team made a special mention for the contribution of Oprimes QA team in their vote of thanks at the final launch event that was attended by who’s who of the film industry. Such appreciation and recognition from their side was commendable considering Oprimes involvement started only at the very last phase of the launch. Interestingly, seeing such wonderful results, crowd testing has now been made a part of their standard release cycle.