Postmates' user experience testing platform

Postmates operates the largest on-demand delivery fleet in 44 major US metropolitan markets. The company’s mission is to power local, on-demand logistics focused on fast deliveries from any type of merchant at scale. The brand has collaborations with Starbucks, Chipotle, 7-Eleven, Walgreens and Apple. Despite all of their success, Postmates’ product team knew that there was room for significant improvement to their app and their three-sided marketplace (restaurants, couriers, and consumers), but were unsure how to identify the changes that would have the largest positive impact in experience.

While they knew that incorporating testing and machine learning would be the best way to glean insight into what their customers really wanted, review of existing experimentation providers proved to be fruitless; many experimentation providers were unable to provide the high level of location and market granularity to solve for things like anonymous experiments and location-based feature flags. Further review also showed that few experimentation vendors were able to function at an equally high level across iOS, Android, and web, and across customer-based and merchant-based experiments. While their marketing stack wasn’t flexible or agile enough to integrate customer data from every touch point, run experiments, and create personalized engagements at scale, the team at Postmates knew that with the right tools, they could build an experimentation platform that would allow them to create incredible customer experiences, no matter the device used and complexity or type of experiment. To build a truly powerful experimentation platform, Postmates decided to create a best-in-class stack that allowed them to move quickly and focus on core competencies, with a CDP as its foundation.

Building a best-in-class stack requires brands to have a clear understanding of exactly what they want to achieve and what their needs are to ensure they create a stack that is not only best-in-class, but best for them. Creating a stack that would enable the level of experimentation they needed to required a foundational customer data layer that was able to integrate with every part of their stack and make data available across the stack to enable:

• Product analytics
• Marketing automation
• Marketing analytics
• User acquisition
• Experimentations
• A/B testing
• Feature flags
• Rolling out
• Attribution
• Return on ad spend

Considering Postmates’ unique needs, they also wanted to ensure that the tools they chose provided a high level of location and market granularity to solve for things like anonymous experiments and location-based feature flags. With this stack in place, the team at Postmates wanted to incorporate experimentation at scale with the goals of using machine learning to:

• Identify precisely where the company was growing, areas of poor growth, and possible future growth areas to guide their product roadmap.
• Get real-time feedback to improve experiments as they roll out
• Understand the qualitative experience for users to improve experiences
• Identify and prioritize the highest ROI initiatives and discover how to ship faster.
• Train their intuition to implement more and better experiments by learning about their products to promote growth.

With all of these requirements in mind, Postmates chose mParticle as the base of their tech stack and the base of their experimentation platform.

Using mParticle, Postmates collects engagement data from on-screen behaviors and engagements and resolves for identity. This instrumentation allows them to understand what customers are doing in the product and how they use it, which makes product analytics easier. Once they identify and understand areas of friction, they leverage their machine learning provider to launch a series of experiments across audience segments to understand which changes will remove friction from these areas.

As new features are rolled out, the Postmates team also builds in experimentation side-by-side, allowing them to evaluate and optimize product features, promotions, and messaging as they go.

One of the first experiments launched by Postmates was focused on reducing friction in their sign up flow. This initial experiment served as a learning experience that set the tone for future experimentation--they chose an opinion about sign up, created a hypothesis, and shipped the first experiment by hard coding it into our iOS app. It was a simple 50/50 test, but it allowed them to understand how customers were using the app. Once they hard coded the experiment, they set exposure, tagged users in treatment and tagged users in control, and monitored the impact on core metrics downstream. By focusing on customers’ sign up experience, rather than a specific outcome, Postmates’ product team was able to show the value of experimentation in motion and truly established a product- and customer-first approach.

Further experimentation around the sign up flow was prompted when the Postmates team discovered a big drop off in one area of sign up. Initially, they hypothesized that the drop off may have been caused by the amount of personal data requested, but found that later in onboarding the dropoff suddenly appeared which invalidated the initial hypothesis. By launching a series of experiments that focused on dropoff during signup and onboarding, they found that not listing a location nearby to new users caused them to drop off. This gave them an initial learning to work off of, but they still needed to quantify how much was really due to being in a city where Postmates wasn’t active. Using this experiment allowed Postmates to get a better understanding of why customers and couriers did or did not complete the sign up, improve the onboarding process,  and could help them to identify high areas of likely growth as defined by dropoff interest.

A second key experiment launched by the Postmates team focused on the effect of the customer feed on purchase and satisfaction. By experimenting with different weighted systems, Postmates pushed favored partner businesses to the top of the feed, creating a much tighter experience. This experiment showed that while customers were likely to purchase from the businesses surfaced in their feed, they also learned that distributions can be changed dramatically as a result of feed manipulation. For instance, trading one partner delivery for another may push some customers to try a favored partner, but it may also push customers to explore the discovery features, which may lead to an inferior experience depending on which vendor is ultimately chosen. When they don’t find something in their feed they want, customers search and go further into their preferences.

When a courier signs up to be a Postmate, they are prompted to take a selfie for identification purposes. Once photos are approved, they are able to begin working as a courier. However, upon initial launch of this feature, they found that users were uploading unclear pictures that led to a 15% image rejection rate. This funnel friction meant that their service area and speed was hindered as viable candidates were rejected. To address this friction, the product team created an on-screen prompt that forced the user to take a new picture in the app. By forcing users to take a new picture rather than uploading from their camera roll, Postmates was able to  dramatically decrease rejection rates, and get new couriers onboarded and working faster.

• Decreased sign up friction to improve new user rates and onboarding process
• Measured and leveraged the impact of customer feeds to improve the quality of deliveries and customer satisfaction
• Improved courier application approval rates, expanding service area and speed
• Created a culture of experimentation
• Improved funnel functions and filtering to foster long tail conversions and disqualify accounts

In the time since Postmates implemented their new best-in-class stack and launched their experimentation engine, they have deployed and learned from thousands of experiments. For the product team at Postmates, using this experimentation engine fueled by individual customer engagement data has allowed them to understand how customers use the product and guide their product roadmap before, during, and after rollout. Experimentation has become a part of the culture at Postmates, allowing for product and marketing teams’ creativity to flourish and launch new experiments to improve customer experiences off the cuff. Using mParticle as their customer data layer provided the foundation they needed to deploy experiments based on clean customer data, analyze customer interactions, and create an integrated stack capable of acting on experiment learnings in real time.

If you'd like to maximize your product output and create customer experiences that exceed expectations across every digital touch point, we'd love to help. Get in touch here to schedule time with one of our experts to start on your AI and ML journey.

mParticle uses AWS to deploy flexible, real-time data solutions reliably to our customers at scale. Using AWS’ large-scaled managed services allows mParticle to track more than 50 billion user sessions and handle over 400 billion API calls per month to help customers power their product, analytics, marketing, and engineering initiatives.