EngineeringAugust 24, 2017

ScyllaDB Migration: How to design high throughput and low latency NoSQL deployments

Yuan Ren, Head of Data Science at mParticle, discusses our ScyllaDB migration and how to process 50 billion monthly messages via NoSQL deployments.

How to design high throughput and low latency NoSQL deployments

mParticle processes and relays incoming messages in near real time, and thus requires a data store that supports querying historical data as well as storing incoming data streams with low latencies. Read on to learn about our ScyllaDB migration and tips for designing high throughput and low latency NoSQL deployments.

mParticle’s ScyllaDB migration:

We initially built the system on top of Apache Cassandra, but as the business grew, we struggled to tune the cluster to keep up with our growing data ingestion rate. Scylla, the next generation of Cassandra, helped us meet our SLA requirements without having to forego the benefits we love from Cassandra. With Scylla’s auto-tuning capability, we can focus our engineering efforts on building products instead of cluster performance.

Check out the corresponding SQL NYC, The NoSQL & NewSQL Database Meetup materials below to learn:

  • How to stop worrying about mix workloads, read-modify-write, compaction and tuning
  • Why mParticle decided on the Apache Cassandra to ScyllaDB migration, an open source database that is fully compatible with Apache Cassandra
  • How to more effectively utilize high capacity Amazon Web Services (AWS) hardware, including the 64 vcpu, 15TB NVMe i3.16xl instances to run a 100TB dataset with just 10 nodes

The mParticle platform captures more than $5 billion in e-commerce transactions over 1 billion monthly active users. Despite its challenging workload, with read-modify-write patterns on huge payload and strict read latency SLAs, we are able to achieve breakneck throughput and low latency without compromising on global database availability….With a little help from our ScyllaDB migration, of course.

Want to solve for the mobile ecosystem’s biggest engineering problems? Learn why you should join the mParticle team and view open roles on our careers page.


Latest from mParticle

MACH Alliance and mParticle featured image


Leading the next generation of CDP solutions: mParticle celebrates acceptance into the MACH Alliance

Madeleine Doyle – April 16, 2024
A stock image of a woman hiking with the onX logo

How onX accelerated marketing campaigns with mParticle's AI engine

April 17, 2024
A headshot of mParticle's Director of Digital Strategy & Business Value, Robin Geier


Introducing Robin Geier: mParticle's Director, Digital Strategy and Business Value

Robin Geier – April 16, 2024
M&S Customer Story

How Marks & Spencer drove revenue growth with mParticle's real-time segmentation and personalization

February 26, 2024

Try out mParticle

See how leading multi-channel consumer brands solve E2E customer data challenges with a real-time customer data platform.

Explore demoSign upContact us

Startups can now receive up to one year of complimentary access to mParticle. Receive access