Why webhooks aren’t enough for rapid deployment
How we resolved to move beyond webhooks making a Firehose SDK mindful of the types of requests and required responses involved in partner integrations.
At mParticle our mission is straightforward: make it really easy for leading apps and app services to connect. Therefore, partner integrations are central to the mParticle platform, and we’re continually optimizing for quality and maintainability of those integrations. Put simply – our partners’ success is our success.
In the early days, all integrations were built in-house, which afforded us the ability to quickly evolve and expand our product, and avoided the challenges related to supporting a publicly available partner API. But, three years and 80+ integrations later, we opened up integration development to our partners by way of our Firehose API last December. Our goal is to make it fast and easy for any SaaS platform to push data to or pull data from our platform.
It’s common across the web to leverage simple webhooks for inter-service communication – they can allow one system to subscribe for changes in another, and that’s incredibly useful. Rather than needing to poll a system for updates, listeners get updates pushed to them if and when they occur. In the case of mParticle, a partner could receive a callback whenever new data to which they’re subscribed arrives in the platform. Though, whereas webhooks seem simple to set up, we found their simplicity would really mean pushing all of the real work onto our partners.
Firehose provides actual models to work with our schema, an SDK encompassing typical integration use-cases, and is designed specifically for rapid deployment:
Publishing our schemaAll partners process data from our platform, which originates as JSON, and translate it into their own structure. To support this, one approach would be to document our API on our doc site and require partners to read and integrate it manually, and update their implementations as we update our API. Instead, we provide actual Java models that are representative of our schema. This means hands-on documentation, compile-time safety, and a clearer way to communicate API updates by way of a versioned SDK.
Bi-directional integrationWith versioned models, our partners have a reliable way to receive and process data, but mParticle integrations need to be bi-directional. For example, when an audience list is created in the mParticle platform, we’ll send that audience to an audience API, such as Facebook Custom Audiences. We then verify that the audience was sent to the partner successfully and continually sync the mParticle audience and the matching partner audience. This sort of tight integration provides a more seamless experience for our customers. It also necessitates that we listen and parse dynamic responses from our partners, which is not a typical design pattern in the context of webhooks.
So, we resolved to move beyond webhooks, and make our Firehose SDK mindful of the types of requests and required responses involved in a partner integration. We defined additional models for the different types of responses for each integration touchpoint, as well as an easy-to-extend framework that exposes hooks for when different types of data are received. This goes as far as having partners programmatically describe the capabilities and requirements of their integration, as well as provide descriptions and even branding – all through our API.
Zero infrastructureFinally, we leverage AWS’s Lambda service to avoid the need for partners to create new infrastructure for their integrations. Rather than deploying new API endpoints and servers to process mParticle data, our partners just write a “lambda function” that implements our SDK. They translate our models into theirs and send data to their existing infrastructure – AWS handles deployment and scaling automatically. A quickstart project with unit tests and a docs tutorial bring it all together.
Firehose – the API, the SDK, and AWS Lambda – is allowing us to add quality integrations at a record pace, and has been crucial in forming a closer relationship with each partner as they’re given ownership over their integrations.
Latest from mParticle
Get your flywheel in motion with Data Master
Learn how mParticle's Data Master enables you to increase data quality throughout the customer data pipeline, allowing insights to compound, and making every campaign and product launch better than the last.
GOAT: Lifecycle marketing for scalable growth
Learn how GOAT uses mParticle to streamline their data pipeline and increase Customer Lifetime Value.
mParticle launches new features to help brands create ‘data flywheel’
New features for seamless data quality management, and transformation to serve as a foundation for improved customer experience and better insights.
Better data, better insights, better results: Helping brands create a data flywheel
Introduce total quality management and enforcement into your customer data pipeline with new Data Master features and Calculated Attributes. With Data Master and Calculated Attributes, establishing a source of reliable customer data that will create a customer data flywheel, where the data quality and data’s impact on the product cycle will continuously improve over time.