Skip to Content
[CAIDA - Center for Applied Internet Data Analysis logo]
Center for Applied Internet Data Analysis > publications : papers : 2019 : hypersparse_neural_network_analysis
Hypersparse Neural Network Analysis of Large-Scale Internet Traffic
J. Kepner, K. Cho, k. claffy, V. Gadepally, P. Michaleas, and L. Milechin, "Hypersparse Neural Network Analysis of Large-Scale Internet Traffic", in IEEE High Performance Extreme Computing Conference (HPEC), Sep 2019.
|   View full paper:    PDF    arXiv    DOI    |  Citation:    BibTeX   |

Hypersparse Neural Network Analysis of Large-Scale Internet Traffic

Jeremy Kepner3
Kenjiro Cho2
kc claffy1
Vijay Gadepally3
Peter Michaleas3
Lauren Milechin3

CAIDA, San Diego Supercomputer Center, University of California San Diego


IIJ Research Lab


Massachusetts Institute of Technology (MIT)

The Internet is transforming our society, necessitating a quantitative understanding of Internet traffic. Our team collects and curates the largest publicly available Internet traffic data containing 50 billion packets. Utilizing a novel hypersparse neural network analysis of “video” streams of this traffic using 10,000 processors in the MIT SuperCloud reveals a new phenomena: the importance of otherwise unseen leaf nodes and isolated links in Internet traffic. Our neural network approach further shows that a two-parameter modified Zipf-Mandelbrot distribution accurately describes a wide variety of source/destination statistics on moving sample windows ranging from 100,000 to 100,000,000 packets over collections that span years and continents. The inferred model parameters distinguish different network streams and the model leaf parameter strongly correlates with the fraction of the traffic in different underlying network topologies. The hypersparse neural network pipeline is highly adaptable and different network statistics and training models can be incorporated with simple changes to the image filter functions.

Keywords: measurement methodology, routing, topology
  Last Modified: Tue Jul-28-2020 14:29:49 UTC
  Page URL: