Big Data is Not Enough.

Fast Data is required to drive real-time personalization.

There is no arguing that Big Data tools like the Hadoop ecosystem have radically changed the technology landscape – but the term “Big Data” is losing useful meaning.

The next frontier in data science is working with Big Data in real-time, so that we can access massive amounts of data instantaneously. Imagine building data products on a platform where time scale is irrelevant, and it’s as easy to retrieve data from years ago as it is from a second ago. An innovation this powerful needs a new name: we call it Fast Data.

As part of the SF Data Mining Meetup series, we hosted a TechTalk where we dove into TellApart’s Predictive Marketing Platform, which is powered by Fast Data. Three hundred Silicon Valley data scientists attended the meetup to hear us describe how we leverage Fast Data to build a diverse set of personalization products on a common infrastructure, and the unique advantages that it gives TellApart in the marketplace.

TellApart’s platform is in part inspired by Nathan Marz’s work describing the Lambda Architecture pattern. Lambda Architecture is a name for a robust, distributed platform that can serve a variety of workloads, including low-latency high-reliability queries. One key insight is that by combining a precomputed batch view of data with a real-time streaming view, you can serve queries in real-time with similar latency to that of a database lookup.

Our focus on building Fast Data systems allows our personalization products to be hyper-responsive to recency. TellApart continues to build a suite of personalization products that can respond to a shopper’s preferences in milliseconds, giving them instantaneous personalization throughout their shopping experience — both onsite as well as offsite. And that personalized shopping experience can include recommendations based on a user profile built over years of shopping behavior, as well as the product that you looked at just a second ago.

In case you missed our event, here’s the video of the TechTalk where you can hear more about how we designed and build our Fast Data infrastructure, as well as where we walk through applying the architecture to drive complex decisions an order of magnitude faster than the blink of an eye.