IODA-NP: Multi-source Realtime Detection of Macroscopic Internet Connectivity Disruption

The focus of this project is on detection and characterization of large-scale Internet outages significatnly affecting large geographic regions or specific Internet operators.

Sponsored by:
Department of Homeland Security (DHS)

Principal Investigators: Alberto Dainotti Marina Fomenkov

Funding source:  70RSAT18CB0000015 Period of performance: July 25, 2018 - January 31, 2020.

Statement of Work

The proposed effort includes applied research, software development, new data analytics, systems integration, operations and maintenance, and an interactive analysis and reporting service. Tasks and deliverables for the entire project are separated into three years:

Year I: Applied Research and Development

Task 1: Definitions and Metrics for Macroscopic Internet Outage Detection
1.1 Define macroscopic Internet outage as the class of Network/Internet scale Disruptive Event (NIDE) targeted for detection
1.2 Illustrate potential causes for this type of event and discuss the rationale behind the approach - NIDE Identification Document (NID)
1.3 Identify quantifiable metrics and the type of sensors and Internet measurement data to use for detecting and characterizing these events
Task 2: IODA: Macroscopic Internet Outage Detection and Analysis Framework
2.1 Develop Internet measurement, data collection, sanitization, aggregation and analysis methodologies to detect the events defined in Task 1 in near-real-time
2.2 Identify the technical and operational capabilities that IODA will provide, as well as the metrics and analytic techniques to be used
2.3 Document the framework - NIDE Analysis Framework Document (NAFD)
Task 3: Macroscopic Internet Outage Analysis, Validation, and Reporting
3.1 Analyze and report Internet outages detected by IODA
3.2 Confirm and validate our inferences using available sources and collaborations with other researchers
3.3 Evaluate the effectiveness of our approach and identify limitations and challenges
Task 4: IODA Interface and Documentation
4.1 Develop Application Programming Interfaces (APIs) to access live stream alerts generated by the system
4.2 Develop visualization interfaces/interactive dashboards for targeted inspection of the observed signals
4.3 Document the service, APIs, and the visual interfaces
Task 5: Continued Support to Software Development, Prepare for Pilot Activities
5.1 Go/No-Go Plan, demonstration, and report
Task 6: Program Management
6.1 Participate in PI meetings
6.2 Quarterly Program Reviews
6.3 Privacy Plan
6.4 Final Report

Milestones and Deliverables (Year I)

# Milestone Deliverable Date Status
1 Technical and Financial Reports Monthly ongoing
2 Program Reviews Quarterly ongoing
3 Kick-off meeting Privacy Plan Aug 25, 2018 done
4 Complete a first version of the NIDE Identification Document Report: NIDE Identification Document (NID) Oct 25, 2018 done
5 Complete a first version of the NIDE Analysis Framework Document Report: NIDE Analysis Framework Document (NAFD) done
6 Purchase and deploy a time series database cluster machine and disk tray done
7 Deploy the BGP based detection approach Nov 25, 2018 done
8 Deploy the active probing based detection approach Jan 25, 2019 done
9 Deploy a first version of the HTTP API done
10 Improve the NIDE Identification Document Report: Updated NID Apr 25, 2019 done
11 Deploy the IBR traffic based detection approach Aug 25, 2019 done
12 Deploy a refined version of the HTTP API Report: NIDE Framework API documentation Oct 25, 2019 done
13 Prepare first NIDE Analysis Report Report: NIDE Analysis Report Nov 25, 2019 done
14 Prepare Go/No-Go evaluation plan Dec 25, 2019 done
15 Deploy the detection fusing all three data sources Software: 1st release of software components (open source) Dec 25, 2019 done
16 Improve the NIDE Analysis Framework Document Report: Updated NAFD Jan 15, 2020 done
17 Go/No Go demonstration Jan 10, 2020 done
18 Report: Final Report Jan 31, 2020 done

Additional Content

IODA-NP: Multi-source Realtime Detection of Macroscopic Internet Connectivity Disruption

The proposal “IODA-NP: Multi-source Realtime Detection of Macroscopic Internet Connectivity Disruption”

Last Modified