Priceline.com A/B Testing Platform

Discovery

The current experimentation process involves 4-5 separate applications and includes:

• Creating a draft test in a complicated web tool
• Notifying engineers and QA through Slack when tests are ready to deploy
• Enrolling in a test through a different web tool
• Monitoring performance through multiple Tableau dashboards
• Sharing results through PowerPoint presentations

User Interviews

In addition to studying the current process, I interviewed data scientists, engineers, product managers, and other possible users on their pain points and needs. With the feedback, I created user journeys, which were vital in informing the design and creating empathy for such specific user types.

Design & Validation

In the first iteration of the design, I focused on the experiment creation and management task flows. On the "Create an Experiment" screen, I reduced the number of required inputs and pages, allowing the user to create an experiment more efficiently. On the "Manage" screen, users can view all experiments and perform quick actions such as edit or restart.

I created an Invision prototype of these designs and tested them with the original interview participants. Using their feedback, I further reduced the amount of information on the screen and iterated on the visual styles. I also expanded the scope of this next iteration to include the experiment monitoring task flow, which allows users to drill down into an individual experiment and view its performance.

Retrospective

If given more time, I would further explore the "Manage" and "Monitor" screens, focusing on the microinteractions of the graphs and tables. Creating a mobile experience for this tool would be valuable, given the feedback that users like to monitor their experiments on-the-go. There were also aspects of the tool that I did not get to, such as the Scorecard, which would enable users to easily share experiment progress with stakeholders.