> Products > Network Management > Lateral Data Processing System
The Achilles Heel of Packet Capture:
What do you do with those billions and trillions of packets?
Why Terabit processing when capturing in Gigabits?
One month encompasses over 43,000 minutes of data capture. For a system that captures and processes data at the same rate, it would take a month to search a month's worth of data. You need a system to be able to process data at a speed several multiples over its capture speed. For example, in one minute a system that captures at 10Gbps and processes at 10Tbps can search through 1000 minutes of captured data (less than a day's worth). One way to boost this ratio is to search just headers, which multiplies the ratio, on average, by about 30, giving you 30,000 minutes of data that can be searched in one minute. This is better, but it still only covers a little over two-thirds of a month of packets in one minute (and is not looking into the payloads). The alternative? Use the sophisticated preprocessing capacity of the Lateral Data Processing. By preconfiguring its 10,000 sorting buckets you can pre-organize the data so that searches, for the most part, would only need to be done on a small subset of the total capture and thereby increase the response speed exponentially. In order to further ensure success you have the aggregation and reporting features to narrow down the scope of searches even further and get even speedier results.
Everyone touts their packet capture speed, trumpeting their 10G and 40G line rates, but they never quite explain what you, in practical terms, do with the data afterwards. It’s the Achilles heel that nobody wants to acknowledge. The former paradigm was that packet capture was the most important element, from which everything else flowed (“Just get your data captured and then you can do anything with it that you want!” That’s what we and others said to our customers ten years ago). But technology moved on. In practicality, if you can’t do anything with the data, there is no use to having the data. As we have found in our decade plus experience with packet capture and Big Data, data retrieval – and we mean timely, fast data retrieval – is key, because that is how you take a huge pile of data and actually make something useful, informative or valuable out of it. Raw packet capture is akin to a cresting river of flood waters, snags and debris – effective and efficient data retrieval takes that overwhelming flood of data and finds the child clinging to a snag within it before it’s too late.
Unless you are working on a network with minimal utilization, quick data retrieval is out of your reach without a rethinking of how it can be achieved. Yesterday’s centralized network appliances with 48 hard drives under one hood simply can’t keep up with the rising tide of data traversing all networks and going in all directions – there is just too huge of an imbalance between data storage and data processing, creating a huge bottleneck. The whole assumption of how to build a system for packet capture and analysis requires a huge re-think, basically taking it out of the box (a large, expensive box) and dividing it among many, smaller boxes. Amazingly, you don’t have to spend a trillion dollars to manage a trillion packets.
Distributed Packet Capture with IPCopper’s Lateral Data Processing
Our Lateral Data Processing system for distributed packet capture and analysis leverages the aggregate power and storage of many to conquer a task that one larger behemoth can’t manage. Our system utilizes three types of machines in a layered approach to capture and distribute the data among many different machines in order to leverage not just their aggregate storage capacity, but also their aggregate processing power. Not only does it capture large amounts of data at high speeds, it delivers large amounts of data to the user at high speeds, whether as reports, aggregates, raw packets or data analyzed and organized in some other fashion.
Lateral Data Processing gives you the ability to divide and conquer the data with aggregate storage capacities of up to tens of thousands of Terabytes (yes, Terabytes) and aggregate processing power measured in Terabits.
For your Lateral Data Processing system we can provide either hardware and software as a bundle or just software. If you source your own hardware, you may realize substantial costs savings over the large 48 HDD behemoths mentioned above by utilizing generic, commodity-type computers, with up to 4 HDDs each, making it possible to have both more storage and more processing power for less cost. While the distributed nature of the system and ability to use commodity hardware get you economy and scale, the robust, proven technology behind Lateral Data Processing brings it all together and helps you make sense of your network and all it contains.
