Project Summary

The UCSD Network Telescope (UCSD-NT) is a passive monitoring system, which captures unsolicited Internet traffic sent to a large segment of unassigned IPv4 address space. For over a decade, this instrumentation has enabled global visibility into macroscopic Internet phenomena that few other data sources can offer. It has provided relevant data used in a broad set of sub-disciplines in Computer & Information Science & Engineering (CISE) and beyond: from network and systems security and stability, to machine learning and big data processing techniques, and, most recently, for studies of cyberwarfare and political repression of communication. In 2011 we enhanced the Telescope instrumentation to enable access to raw and live telescope traffic data, thus expanding the scope of possible research questions and the circle of researchers using the data. As of January 2017 we were aware of over 100 publications (a lower bound) - without UCSD co-authors - that used UCSD-NT data. Yet the infrastructure was lagging behind the increasing demands in terms of storage, computing resources, and system administration. These issues hindered our ability to continue sharing UCSD-NT data with researchers, and required compromises that limited the availability of this unique resource.

The STARDUST project will help extend and sustain operation of the UCSD-NT infrastructure. We will upgrade and modernize the current infrastructure to handle the predicted growth in traffic, leverage virtualization and NSF-funded HPC platforms at the San Diego Supercomputer Center for computational data analysis, and introduce meta-data semantics to simplify many tasks researchers typically want to do with UCSD-NT data. The proposed modifications will leave researchers more time (and available HPC resources) to focus on their specific scientific questions. Moreover, the project will forge an interdisciplinary collaboration between researchers from the field of computer networks and HPC scientists and engineers to experiment with novel approaches for research on live traffic analysis.

The stabilized and enhanced infrastructure capabilities will better serve a diverse range of academic researchers, the vast majority of whom have no access to any other source of global Internet traffic data. The proposed enhancements will support invaluable hands-on experience in operationally relevant network security and traffic analysis research engaging a wide audience of computer science faculty and students in the use of our tools and data. Project results will contribute to advancing knowledge in diverse CISE disciplines, e.g., facilitating the development of efficient strategies for early detection and mitigation of cyber attacks, supporting macroscopic Internet performance and reliability assessments, and opening a new domain for the application of live streaming big data analysis and in situ machine learning techniques.

Project Milestones

•  Task 1 : Upgrade and modernize the UCSD-NT infrastructure (Years 1 and 2);
•  Task 2 : Transition the data analysis infrastructure to use NSF HPC resources (Years 1, 2, and 3);
  •  Task 2.1 : Deploy cloud-compute support using novel virtualization features on Comet supercomputer (Years 1 and 2);
  •  Task 2.2 : Develop and deploy live packet capture and distribution software (Years 1 and 2);
  •  Task 2.3 : Support dynamically provisioned specialized Big Data environments (Years 1 and 2);
•  Task 3 : Reduce processing complexity and simplify data analysis (Years 1, 2, and 3).
•  Task 4 : Communal activities (Years 1, 2, and 3).

Project Timeline