Skip to Content
[CAIDA - Center for Applied Internet Data Analysis logo]
Center for Applied Internet Data Analysis
A Robust System for Accurate Real-time Summaries of Internet Traffic
K. Keys, D. Moore, and C. Estan, "A Robust System for Accurate Real-time Summaries of Internet Traffic", in SIGMETRICS, Jun 2005, pp. 85--96.
|   View full paper:    PDF    |  Citation:    BibTeX   |

A Robust System for Accurate Real-time Summaries of Internet Traffic

Ken Keys1
David Moore1
Cristian Estan2
1

CAIDA, San Diego Supercomputer Center, University of California San Diego

2

Computer Sciences Department, University of Wisconsin-Madison

Good performance under extreme workloads and isolation between the resource consumption of concurrent jobs are perennial design goals of computer systems ranging from multitasking servers to network routers. In this paper we present a specialized system that computes multiple summaries of IP traffic in real time and achieves robustness and isolation between tasks in a novel way: by automatically adapting the parameters of the summarization algorithms. In traditional systems, anomalous network behavior such as denial of service attacks or worms can overwhelm the memory or CPU, making the system produce meaningless results exactly when measurement is needed most. In contrast, our measurement system reacts by gracefully degrading the accuracy of the affected summaries.

The types of summaries we compute are widely used by network administrators monitoring the workloads of their networks: the ports sending the most traffic, the IP addresses sending or receiving the most traffic or opening the most connections, etc. We evaluate and compare many existing algorithmic solutions for computing these summaries, as well as two new solutions we propose here: "flow sample and hold" and "Bloom filter tuple set counting". Compared to previous solutions, these new solutions offer better memory versus accuracy tradeoffs and have more predictable resource consumption. Finally, we evaluate the actual implementation of a complete system that combines the best of these algorithms.

Keywords: passive data analysis
  Last Modified: Wed Oct-11-2017 17:03:52 PDT
  Page URL: http://www.caida.org/publications/papers/2005/flowest/index.xml