Skip to Content
[CAIDA - Center for Applied Internet Data Analysis logo]
Center for Applied Internet Data Analysis > funding : impact-assists
ASSISTS - Advancing Scientific Study of Internet Security and Topological Stability
Sponsored by:
Department of Homeland Security (DHS)

CAIDA participates in the Information Marketplace for Policy and Analysis of Cyber-risk & Trust (IMPACT) program as a Data Provider and as a Decision Analytics-as-a-Service Provider.

Funding source: DHS FA8750-18-2-0049. Period of performance: December 18, 2017 - September 17, 2019.

Project Summary

Large-scale Internet cyber-attacks and incidents - route hijacking, network outages, fishing campaigns, botnet activities, large-scale bug exploitation, etc. - represent a major threat to public safety and to both public and private strategic and financial assets. Mitigation and recovery, assessment of impacts and restoration costs, as well as prevention of further attacks of similar nature, are impeded by the fact that such events can remain unnoticed or are hard to characterize, in terms of motivation, infrastructure used by the attacker, and scope. Because of their macroscopic nature, identifying such events and understanding their scope and dynamics requires three critical inputs:

  • heterogeneous sources and types of data to cross-validate inferences;
  • a system to enable close to real-time integration and interactive visualization of such data;
  • a team of experts with varied background and skills to soundly interpret fused data.

We are pursuing these three inputs via strategically planned two-fold participation in the IMPACT program. As a Data Provider, we will continue to provide data sets that have already proven relevant to researchers studying security, stability, and resilience of networks. As a Decision Analytics-as-a-Service Provider, we will support new analytic capabilities that integrate, correlate, and cross-validate multiple sources of measurement and meta-data to enable informed mitigation of and response to attacks and other disruptive events.

Statement of Work

CAIDA performs fundamental research on a reasonable efforts basis and in accordance with UC policy. Technical reports will be submitted triannually.

TTA #1: Supporting Cybersecurity Research through Network Data Collection and Curation

SubtaskDescriptionProjected TimelineStatus
Data Provider Tasks
1Curate and package the Internet Topology Measured from Ark Platform datasetsongoing
2Curate and package the Internet Topology Data Kitsevery 3-6 mo
3Curate and package the UCSD Real-time Network Telescope Datasetsongoing
4Collect, process, and archive the U.S. backbone bidirectional traffic data*
*as long as conditions permit and links and traffic monitors are available
5Acquire a 100gb packet capture monitorYear 2
6Deploy the packet capture monitor on a 100gb national backbone linkYear 2
Data Host Tasks
1Maintain and expand our hosting capabilitiesongoing
2Manage, maintain, and serve previously collected CAIDA dataongoing
3Index and share new CAIDA data sets with researchersongoing
4Compile statistics of data volumes, requests and downloadongoing
New Data Sets
1Generate new data sets that are crucial for studying threats, vulnerabilities, and hazards to critical infrastructuresongoing
2Generate derivative data sets that reveal signals of connectivity disruptions from active and passive measurement methodsYear 2
3Experiment with which possible data sets are most amenable to live streaming to support HI-CUBE's near-real-time analytic capabilitiesYear 2
Project Support
1Work closely with other IMPACT project team membersongoing
2Work closely with IMPACT Portal developersongoing
3Update IMPACT MOAs to support new data offeringsas needed
4Host and attend project meetingsas needed
5Provide documentation, outreach materials, marketing effortsongoing


1Hosting Infrastructure DescriptionAnuallyApr 2018
2Summary of use and utility of CAIDA's IMPACT dataAnnually

TTA #2: Developing HI-CUBE: Hub for Internet Incident Investigation

SubtaskDescriptionProjected TimelineStatus
Development of web services and visual interfaces
1Extend the authorization functionality of the current Charthouse web application to support fine-grained data access controlYear 1
2Develop a management interface for users, groups and shared dataYear 2
Design and development of software infrastructure for data storage, query, and transformation
1Replace our monolithic time series database (DBATS) with a distributed database for time-series analytics (e.g. Apache Kudu, Influx DB Enterprise version)Year 2
2Replace the Graphite back-end that queries DBATS with a Big Data analytics query engine (based on Apache Spark and Spark SQL)Year 2
Integration and testing of HI-CUBE system in operational research environments
1Acquire the hardware needed for hosting the serviceYear 1
2Migrate current databases and integrate additional datasets developedYear 1
3Deploy, benchmark, and tune the upgraded components of the infrastructure for big data analyticsMar 2019
4Migrate the time series currently stored in DBATS into the new systemMar 2019
5Deploy the query engine and the HTTP query serverAug 2019
6Integrate and test the big data analytics query engineYear 2
Community outreach and service
1Collect feedback during meetings and presentationsongoing
2Interact with the users of the platform to better focus our efforts to better serve the community of cybersecurity researchers and analystsongoing
3Present the HI-CUBE platform in one or more of our CAIDA workshops---


1Deploy SSD cluster machine, storage server, and disk trayJun 2018done
2Deploy Web Application ServerSep 2018
3Extend the authentication and authorization functionalitySep 2018
4Release alpha version of prototype websiteSep 2018done
5Migrate time series currently stored in DBATSMar 2019
6Deploy second Web Application ServerMar 2019
7Complete the development of a distributed database for time-series analyticsMar 2019
8Develop management interfaces for users, groups and shared dataMar 2019
9Deploy second SSD cluster machineMay 2019
10Complete the tuning of the distributed database systemJul 2019
11Deploy the query engine and the HTTP query serverAug 2019
12Complete the development of the Data analytics query engineSep 2019
13Release beta version of prototype web siteSep 2019
14Release as open source the distributed time-series database and query engineSep 2019


1Capability Design PlanFeb 2018done
2Demonstrate web service at PI MeetingsTriannually
3Open source HI-CUBE software packageSep 2019

Acknowledgement of awarding agency's support

This material is based on research sponsored by Air Force Research Laboratory under agreement number FA8750-18-2-0049. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of Air Force Research Laboratory or the U.S. Government.

  Last Modified: Mon May-21-2018 13:15:07 PDT
  Page URL: