We need better numbers. Or not?

Problem

Replai uses computer vision AI to aggregate creative video data and to tag video elements at scale to unlock new actionable insights for marketing teams. It’s used by the big players in the gaming industry, especially hyper-casual games, but also by other industries such as the recent big data influencers agencies and more.

Before UX Research was established in the company, Replai already had the drive to collect user feedback from different sources: analytics, customer experience operatives, product cold calls with key customers, and more. But all this knowledge was skewed and a bit siloed. Because everyone in Replai has a very scientific approach to everything, they were aware of this data biasing and wanted to collect user-related data and feedback that was more robust and significant.

Each one of the them had a different view on which could be the present issues and opportunities, but they shared an opinion that if they were able to deliver more accurate data their users and deciders (those who pay for the tool) would value the platform more.

What was done

First things, first. My first step was to collect all existing user feedback from past 2 years. This included 40+ video meetings recordings, slack messages from 20+ channels, recordings from Dovetail, and more. Next, I process all this data using qualitative analysis + Affinity Diagramming to find patterns of behaviors and attitudes. These patterns and respective link to each related content was further organised in a new Dovetail area called User Feedback Hub.

Just by doing this, I found various issues and opportunities to improve Replai’s platform. But that was not enough, since the information was not collected with a robust experimental design or best practices of research. It was just a first source of information. To get a robust set of data I need to triangulate with other sources of information, and preferably qual and quant data.

I conducted an exploratory contextual inquiry to re-discover new things about current top and top adjacent users and to explore the product gaps that could signify a new opportunity space. The final top users were the ones existing in the Mix Panel Top User cohort and super active on Hotjar recordings; users super active on Hotjar only (not on Mix Panel) were considered adjacent top users. Furthermore, I created a survey as a quick form of feedback about specific product inquiries, but its results also provided information for users’ product expectations. This survey contained two feature ranking questions and two other questions about specific product ideas. In total, I triangulated 5 sources of information to find the most critical issues and opportunities for Replai’s business and product.

The contextual inquiries were conducted with 8 top users from different companies and with varying roles (Creative, marketing, UA) and levels of seniority (lead, specialist, manager). The whole research endeavor, from collecting to processing and delivering results took 2 months, primarily due to the difficulty of getting free time from top users to spend one hour with me. I also used MixPanel to understand platform usage throughout time: weekly, daily, and hourly usage to cross-check with contextual inquiry findings.

Impact

It is said that a good research shouldn’t deliver a lot of results; it should deliver quality, digestible and actionable ones. And this was such a case. For the first time, product and business felt secure about their decisions, because they were armed with robust knowledge about their users. Actionable knowledge such as:

• User journey maps for each top task
• Users weekly work goals, and all the processes they do to achieve those, including common challenges
• Other tools and platforms users use to get to those weekly goals – this drove a large competitor analysis on the product side
• Replai’s platform usage crossed with this external user goals
• Cognitive walkthrough map for the different user profiles, including their decision points and related pain points in their real day to day goals
• Cognitive walkthrough map for users interacting with the platform, with and without tasks assigned to them
• Most important one: our main user was a Creator, not an Analyst, as we thought before. Creators really needed to produce more creatives each week and find winners fast. They needed help to create more and with quality. I concluded that this perception of better data being the best strategy was because the users who stated that need were just louder than the rest. But they were a minority. The majority of users were creators and needed better tools to create. So, Replai started to work on a project of a new area where AI would make combinations of clients’ ad elements considering a user prompt and all the data on the best-performing elements. This was before MidJourney and the AI boom!