Outcomes Report

Public and private sector stakeholders around the world seek ways to ensure that the Internet provides the level of reliability and resilience we have long taken for granted from the telephony network. Unfortunately, in spite of the societal and economic impact of Internet connectivity disruptions, we lack near-realtime, scalable and validated methodologies and tools to identify and understand large-scale Internet outages.

Based on experimental work in which we combined measurements at the control plane, active probing and passive traffic analysis, CAIDA developed an operational prototype system, IODA (Internet Outage Detection and Analysis), that monitors the Internet, in near-realtime, with the goal of identifying macroscopic Internet outages affecting the edge of the network, i.e., significantly impacting an AS or a large fraction of a country. One of the first research platforms offering a multi-source view of large-scale Internet outages, IODA enabled us to design, create, deploy, and support powerful instrumentation for outage detection and analysis, This project spurred several collaborations with researchers to experiment with integration of new detection methodologies, and improve our own. Partners have included the U.S. Federal Communications Commission, and a large Internet broadband service provider (ISP) who already uses IODA to support their own research on Internet reliability and performance.

During this project, we reported and documented several relevant episodes of connectivity disruption, including outages affecting large U.S. residential broadband ISPs, as well as entire countries almost entirely disconnected from the Internet (e.g., North Korea, Sudan, Syria). We published 10 research papers, and gave more than 20 presentations in prestigious international venues with participation of industrial and academic communities (IETF, NANOG, RIPE, ACM IMC, etc.) as well as in meetings with government agencies. We released the software components we developed in this project with an open source license. Several of the project's software components offer general applicablity to research and applicative fields of networking, in addition to providing critical functionality to the IODA system. For example, BGPStream, one of the flagship software platforms developed for IODA, delivers a framework that has filled a large gap in the availability of efficient tools and programmer interfaces for the analysis and monitoring of Internet global routing. Researchers and practitioners around the world already use BGPStream for a range of historical and real-time BGP data analysis applications. The codebase receives contributions from third parties and has spurred a new collaboration with industry to extend the platform's compatibility, and to support novel Internet monitoring technologies.

To understand the health and resiliency of the global Internet we need tools that synthesize and cross-correlate systematic and targeted active probing measurements from globally distributed vantage points, with new techniques for mining passively collected data sources to make efficient and intelligent use of extracted signals of critical infrastructure outages. We also need dashboards and information visualization methods to deliver near real-time notification and nimble interactive visual displays for deeper analysis and targeted querying. The IODA system instantiates a vision for such a system, and contributes fundamental open-source components of the knowledge pipeline that can serve as building blocks for future systems. The project has also seeded a valuable collaboration between academia and industry to validate the evolving methodologies against ground truth.