Skip to Content
[CAIDA - Center for Applied Internet Data Analysis logo]
Center for Applied Internet Data Analysis > funding : impact-assists
ASSISTS - Advancing Scientific Study of Internet Security and Topological Stability
Sponsored by:
Department of Homeland Security (DHS)

CAIDA participates in the Information Marketplace for Policy and Analysis of Cyber-risk & Trust (IMPACT) program as a Data Provider and as a Decision Analytics-as-a-Service Provider.

Funding source: DHS cooperative agreement FA8750-18-2-0049. Period of performance: December 18, 2017 - September 17, 2019; September 18, 2019 - May 31, 2020 (no cost extension).

Project Summary

Large-scale Internet cyber-attacks and incidents - route hijacking, network outages, fishing campaigns, botnet activities, large-scale bug exploitation, etc. - represent a major threat to public safety and to both public and private strategic and financial assets. Mitigation and recovery, assessment of impacts and restoration costs, as well as prevention of further attacks of similar nature, are impeded by the fact that such events can remain unnoticed or are hard to characterize, in terms of motivation, infrastructure used by the attacker, and scope. Because of their macroscopic nature, identifying such events and understanding their scope and dynamics requires three critical inputs:

  • heterogeneous sources and types of data to cross-validate inferences;
  • a system to enable close to real-time integration and interactive visualization of such data;
  • a team of experts with varied background and skills to soundly interpret fused data.

We are pursuing these three inputs via strategically planned two-fold participation in the IMPACT program. As a Data Provider, we will continue to provide data sets that have already proven relevant to researchers studying security, stability, and resilience of networks. As a Decision Analytics-as-a-Service Provider, we will support new analytic capabilities that integrate, correlate, and cross-validate multiple sources of measurement and meta-data to enable informed mitigation of and response to attacks and other disruptive events.

Statement of Work

CAIDA performs fundamental research on a reasonable efforts basis and in accordance with UC policy. Technical reports will be submitted triannually.

TTA #1: Supporting Cybersecurity Research through Network Data Collection and Curation

SubtaskDescriptionProjected TimelineStatus
1. Data Provider Tasks
1.1Curate and package the Internet Topology Measured from Ark Platform datasetsongoingArk topology datasets indexed in IMPACT
1.2Curate and package the Internet Topology Data Kitsevery 3-6 moITDK CAIDA page
1.3Curate and package the UCSD Real-time Network Telescope DatasetsongoingTelescope datasets indexed in IMPACT
1.4Collect, process, and archive the U.S. backbone bidirectional traffic data*
*as long as conditions permit and links and traffic monitors are available
ongoingAnonymous Internet Traces Dataset
1.5Acquire a 100gb packet capture monitorYear 2
1.6Deploy the packet capture monitor on a 100gb national backbone linkYear 2
2. Data Host Tasks
2.1Maintain and expand our hosting capabilitiesongoingSize of datasets Indexed in IMPACT
2.2Manage, maintain, and serve previously collected CAIDA dataongoing
2.3Index and share new CAIDA data sets with researchersongoingCAIDA Data Overview Table
2.4Compile statistics of data volumes, requests and downloadongoingIMPACT datasets access requests stats
3. New Data Sets
3.1Generate new data sets that are crucial for studying threats, vulnerabilities, and hazards to critical infrastructuresongoingList of new datasets indexed in IMPACT
3.2Generate derivative data sets that reveal signals of connectivity disruptions from active and passive measurement methodsYear 2
3.3Experiment with which possible data sets are most amenable to live streaming to support HI-CUBE's near-real-time analytic capabilitiesYear 2
4. Project Support
4.1Work closely with other IMPACT project team membersongoing
4.2Work closely with IMPACT Portal developersongoing
4.3Update IMPACT MOAs to support new data offeringsas needed
4.4Host and attend project meetingsas neededDHS IMPACT PI Meetings/Presentations
4.5Provide documentation, outreach materials, marketing effortsongoingList of Outreach Publications/Presentations


1Hosting Infrastructure DescriptionAnuallyApr 2018
2Summary of use and utility of CAIDA's IMPACT dataAnnuallyTri-annual Reports

TTA #2: Developing HI-CUBE: Hub for Internet Incident Investigation

SubtaskDescriptionProjected TimelineStatus
1. Development of web services and visual interfaces
1.1Extend the authorization functionality of the current Charthouse web application to support fine-grained data access controlYear 1done
1.2Develop a management interface for users, groups and shared dataYear 2
2. Design and development of software infrastructure for data storage, query, and transformation
2.1Replace our monolithic time series database (DBATS) with a distributed database for time-series analytics (e.g. Apache Kudu, Influx DB Enterprise version)Year 2ongoing
2.2Replace the Graphite back-end that queries DBATS with a Big Data analytics query engine (based on Apache Spark and Spark SQL)Year 2
3. Integration and testing of HI-CUBE system in operational research environments
3.1Acquire the hardware needed for hosting the serviceYear 1done
3.2Migrate current databases and integrate additional datasets developedYear 1done
3.3Deploy, benchmark, and tune the upgraded components of the infrastructure for big data analyticsYear 2
3.4Migrate the time series currently stored in DBATS into the new systemYear 2
3.5Deploy the query engine and the HTTP query serverYear 2
3.6Integrate and test the big data analytics query engineYear 2
4. Community outreach and service
4.1Collect feedback during meetings and presentationsongoing
4.2Interact with the users of the platform to better focus our efforts on the needs of the community of cybersecurity researchers and analystsongoing
4.3Present the HI-CUBE platform in one or more of our CAIDA workshopsMay 2020done


1Deploy SSD cluster machine, storage server, and disk trayJun 2018done
2Deploy Web Application ServerSep 2018done
3Extend the authentication and authorization functionalitySep 2018done
4Release alpha version of prototype websiteSep 2018done
5Migrate time series currently stored in DBATSMar 2019
6Deploy second Web Application ServerMar 2019ongoing
7Complete the development of a distributed database for time-series analyticsMar 2019ongoing
8Develop management interfaces for users, groups and shared dataMar 2019
9Deploy second SSD cluster machineMay 2019ongoing
10Complete the tuning of the distributed database systemJul 2019
11Deploy the query engine and the HTTP query serverAug 2019
12Complete the development of the Data analytics query engineMay 2020
13Release beta version of prototype web siteMay 2020
14Release as open source the distributed time-series database and query engineMay 2020


1Capability Design PlanFeb 2018done
2Demonstrate web service at PI MeetingsTriannuallyongoing
3Open source HI-CUBE software packageMay 2020

Acknowledgement of awarding agency's support

This material is based on research sponsored by Air Force Research Laboratory under agreement number FA8750-18-2-0049. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the U.S. Government.

  Last Modified: Tue Nov-26-2019 12:18:52 PST
  Page URL: