Parse.ly processes a lot of data. As a matter of fact, we store 170+ terabytes of compressed analytics data across all of our customer accounts, a number that is currently growing at 20- to 30-terabytes per month.
We thought it would be interesting to share some additional background to give you a better sense of what is going on behind the scenes:
As you can see, Parse.ly is currently tracking about 50 billion monthly user actions (e.g. pageviews, mobile views, long scrolls/reads) for our customer websites. To put that number into perspective: 50 billion seconds is equivalent to 1,500 years.
Parse.ly is aggregating data from more than 475 million unique web visitors per month. Did you know that the population of the United States of America is currently 318 million?
Our time series engine, Mage, stores 12 terabytes of live query data on solid-state storage (SSDs) and stores 1.2 terabytes of live memory (RAM) for caching and indexing. For context, that’s the same as you might find in about 150 modern Apple Macbook Pro computers.
Parse.ly stores a lot of data because our customers send us a lot of data, and they want answers from all that data. Because Parse.ly does not sample its data set and analyzes complex behavior — like views, visitors, engaged time, and social shares — for complex websites (each website potentially has thousands or millions of distinct URLs), we have to store a lot of data to make sense of any of it.
Our customers run the biggest media websites across the web, and we have lots of customers, so it all adds up!