IODA-NP: Multi-source Realtime Detection of Macroscopic Internet Connectivity Disruption

The focus of this project is on detection and characterization of large-scale Internet outages significatnly affecting large geographic regions or specific Internet operators.

Statement of Work

The proposed effort includes applied research, software development, new data analytics, systems integration, operations and maintenance, and an interactive analysis and reporting service. Tasks and deliverables for the entire project are separated into three years:

Year I: Applied Research and Development


Task 1: Definitions and Metrics for Macroscopic Internet Outage Detection
1.1	Define macroscopic Internet outage as the class of Network/Internet scale Disruptive Event (NIDE) targeted for detection
1.2	Illustrate potential causes for this type of event and discuss the rationale behind the approach - NIDE Identification Document (NID)
1.3	Identify quantifiable metrics and the type of sensors and Internet measurement data to use for detecting and characterizing these events
Task 2: IODA: Macroscopic Internet Outage Detection and Analysis Framework
2.1	Develop Internet measurement, data collection, sanitization, aggregation and analysis methodologies to detect the events defined in Task 1 in near-real-time
2.2	Identify the technical and operational capabilities that IODA will provide, as well as the metrics and analytic techniques to be used
2.3	Document the framework - NIDE Analysis Framework Document (NAFD)
Task 3: Macroscopic Internet Outage Analysis, Validation, and Reporting
3.1	Analyze and report Internet outages detected by IODA
3.2	Confirm and validate our inferences using available sources and collaborations with other researchers
3.3	Evaluate the effectiveness of our approach and identify limitations and challenges
Task 4: IODA Interface and Documentation
4.1	Develop Application Programming Interfaces (APIs) to access live stream alerts generated by the system
4.2	Develop visualization interfaces/interactive dashboards for targeted inspection of the observed signals
4.3	Document the service, APIs, and the visual interfaces
Task 5: Continued Support to Software Development, Prepare for Pilot Activities
5.1	Go/No-Go Plan, demonstration, and report
Task 6: Program Management
6.1	Participate in PI meetings
6.2	Quarterly Program Reviews
6.3	Privacy Plan
6.4	Final Report

Milestones and Deliverables (Year I)

#	Milestone	Deliverable	Date	Status
1		Technical and Financial Reports	Monthly	ongoing
2	Program Reviews		Quarterly	ongoing
3	Kick-off meeting	Privacy Plan	Aug 25, 2018	done
4	Complete a first version of the NIDE Identification Document	Report: NIDE Identification Document (NID)	Oct 25, 2018	done
5	Complete a first version of the NIDE Analysis Framework Document	Report: NIDE Analysis Framework Document (NAFD)		done
6	Purchase and deploy a time series database cluster machine and disk tray			done
7	Deploy the BGP based detection approach		Nov 25, 2018	done
8	Deploy the active probing based detection approach		Jan 25, 2019	done
9	Deploy a first version of the HTTP API		Jan 25, 2019	done
10	Improve the NIDE Identification Document	Report: Updated NID	Apr 25, 2019	done
11	Deploy the IBR traffic based detection approach		Aug 25, 2019	done
12	Deploy a refined version of the HTTP API	Report: NIDE Framework API documentation	Oct 25, 2019	done
13	Prepare first NIDE Analysis Report	Report: NIDE Analysis Report	Nov 25, 2019	done
14	Prepare Go/No-Go evaluation plan		Dec 25, 2019	done
15	Deploy the detection fusing all three data sources	Software: 1st release of software components (open source)	Dec 25, 2019	done
16	Improve the NIDE Analysis Framework Document	Report: Updated NAFD	Jan 15, 2020	done
17		Go/No Go demonstration	Jan 10, 2020	done
18		Report: Final Report	Jan 31, 2020	done

Additional Content

IODA-NP: Multi-source Realtime Detection of Macroscopic Internet Connectivity Disruption

The proposal “IODA-NP: Multi-source Realtime Detection of Macroscopic Internet Connectivity Disruption”