Priceline.com A/B Testing Platform
Deliverables: User Interviews, Journey Maps, Web Wireframes, Web Prototype, Usability Testing
Tools: Sketch, InvisionApp, Zeplin
tl;dr
Product growth at Priceline.com is centered around A/B testing, but the current process requires product managers, engineers, and others to use disjointed tools and perform manual tasks. To improve efficiency and test accuracy, the data science team decided to build a custom application and asked me to lead the design.
Discovery
The current experimentation process involves 4-5 separate applications and includes:
- Creating a draft test in a complicated web tool
- Notifying engineers and QA through Slack when tests are ready to deploy
- Enrolling in a test through a different web tool
- Monitoring performance through multiple Tableau dashboards
- Sharing results through PowerPoint presentations
Tools in current process
User Interviews
In addition to studying the current process, I interviewed data scientists, engineers, product managers, and other possible users on their pain points and needs. With the feedback, I created user journeys, which were vital in informing the design and creating empathy for such specific user types.
Design & Validation
In the first iteration of the design, I focused on the experiment creation and management task flows. On the "Create an Experiment" screen, I reduced the number of required inputs and pages, allowing the user to create an experiment more efficiently. On the "Manage" screen, users can view all experiments and perform quick actions such as edit or restart.
Current create experiment screen
Redesigned create experiment screen
Manage screen
I created an Invision prototype of these designs and tested them with the original interview participants. Using their feedback, I further reduced the amount of information on the screen and iterated on the visual styles. I also expanded the scope of this next iteration to include the experiment monitoring task flow, which allows users to drill down into an individual experiment and view its performance.
Manage screen v2
Monitor screen
Monitor - experiment details
Retrospective
If given more time, I would further explore the "Manage" and "Monitor" screens, focusing on the microinteractions of the graphs and tables. Creating a mobile experience for this tool would be valuable, given the feedback that users like to monitor their experiments on-the-go. There were also aspects of the tool that I did not get to, such as the Scorecard, which would enable users to easily share experiment progress with stakeholders.
Back to top