RSDoS - STARDUST
RSDoS
Overview
Observing Denial-of-service Attacks through Network Telescopes
Network telescopes can be used to detect and observe the randomly-spoofed distributed denial-of-service (RSDoS) attacks happening worldwide. To make it difficult for the attack victim (and the victim’s ISPs) to block an incoming attack, the attacker may use a fake source IP address (similar to a fake return address in postal mail) in each packet sent to the victim. Because the DoS attack victim can’t distinguish between incoming requests from an attacker and legitimate inbound requests, the victim tries to respond to every received request. When the attacker spoofs a source address in the network telescope, we observe a response destined for a computer that doesn’t exist (and therefore never sent the initial query). By detecting these unsolicited responses, researchers can identify DoS attack victims and infer information about the volume of the attack, the bandwidth of the victim, the location of the victim, and the types of services the attacker targets.
Definition
This dataset contains meta-data of randomly spoofed DoS attacks from the backscatter packets collected by the UCSD Network Telescope. The data is aggregated from the raw telescope data and is updated every day, containing data starting from October 1, 2008. The data is generated by processing 5-minute intervals of raw telescope data and extracting the response packets sent by victims of RSDoS attacks.
The criteria used to aggregrate the data is outlined in this paper: Inferring Internet Denial-of-Service Activity (2006) by Moore et al.
DoS Data
The RSDoS-Corsaro3 plugin attempts to port the data collection method from Corsaro 2 over to the new Corsaro 3 framework. However, due to increasing traffic volumes (and therefore increasing processing requirements), we have had to make some minor changes to the structure of the data that we save.
Definition
Properties
RSDoS-Corsaro3 writes its output as files using the Apache Avro format. Each record in an Avro files describes an individual attack that was observed within a particular 5 minute time interval. Attacks that span multiple time intervals will write a record for each interval that they occur in.
Record Format
| Property | Data Type | Description |
|---|---|---|
| bin_timestamp | long | The timestamp for the interval that this attack was observed in. |
| initial_packet_len | int | The size of the first packet observed as part of this attack. |
| target_ip | long | The IP address of the address that was the target of the DoS attack (i.e. the source address of the observed packets). Encoded as a 32 bit integer. |
| target_protocol | int | The transport protocol used for the attack (1 = ICMP, 6 = TCP, 17 = UDP). |
| attacker_slash16_cnt | long | The number of distinct /16 subnets in our monitored network that received packets from the victim. |
| attack_port_cnt | long | The number of unique source ports used by the attacker (i.e. the number of unique destination ports seen on received packets attributed to this attack). |
| target_port_cnt | long | The number of unique ports that were targeted on the victim (i.e. the number of unique source ports seen on received packets attributed to this attack). |
| packet_cnt | long | The number of packets that were attributed to this attack. |
| icmp_mismatches | long | The number of ICMP packets attributed to this attack where the source IP address in the body of the ICMP packet (e.g. the original datagram reflected in a Destination Unreachable message) does not match the IP address that the ICMP packet was sent to. |
| byte_cnt | long | The number of bytes that have been sent to our network due to this attack (based on IP length). |
| max_ppm_interval | long | The peak observed packet rate observed for this attack. |
| start_time_sec | long | The seconds portion of the Unix timestamp of the first packet attributed to this attack. |
| start_time_usec | int | The microseconds portion of the Unix timestamp of the first packet attributed to this attack. |
| latest_time_sec | long | The seconds portion of the Unix timestamp of the last packet attributed to this attack. |
| latest_time_usec | int | The microseconds portion of the Unix timestamp of the last packet attributed to this attack. |
| first_attack_port | int | The source port that was used by the first packet that was attributed to this attack. |
| first_target_port | int | The destination port that was used by the first packet that was attributed to this attack. |
| maxmind_continent | string | The continent where the target IP address is located, according to Maxmind geo-location data. |
| maxmind_country | string | The country where the target IP address is located, according to Maxmind geo-location data. |
| initial_packet | bytes | A binary blob containing the entire contents of the first packet attributed to this attack, including link layer headers. |
User Guide
RSDoS data collected using corsaro3 is stored in Openstack Swift using the Apache Avro data format.
RSDoS data collected for the UCSD network telescope can be found in Swift within
the telescope-ucsdnt-avro-rsdos container.
Users can read the Avro files directly using existing Avro-compatible tools or libraries, such as the official Apache Avro tools .jar file or the avrocat tool that is available in PyPi.
For writing your own analysis tools, a better option would be to use the
PyAvro-STARDUST module to
read RSDoS data within Python scripts, which will be faster and simpler to
use than using fastavro or other more generic Python libraries. An example
script for processing RSDoS data can be found here.
Example using avrocat
limbo@username:~$ wandiocat swift://telescope-ucsdnt-avro-rsdos/datasource=ucsd-nt/year=2020/month=07/day=14/hour=23/ucsd-nt.1594770900.rsdos.avro | avrocat | head
{"bin_timestamp":1623796800,"initial_packet_len":86,"target_ip":2412812948,"target_protocol":6,"attacker_slash16_cnt":171,"attack_port_cnt":570,"target_port_cnt":565,"packet_cnt":570,"icmp_mismatches":0,"byte_cnt":39948,"max_ppm_interval":90,"start_time_sec":1623796672,"start_time_usec":584850,"latest_time_sec":1623797099,"latest_time_usec":667525,"first_attack_port":35598,"first_target_port":50474,"maxmind_continent":"SA","maxmind_country":"BR","initial_packet":"<ýþ\u0019Ø\u0000\u0000Þûº\u0006Ç\b\u0000EÈ\u0000D|e\u0000\u00002\u0001=Ö\u0000x\u0000,.\u000B\u0000æ4\u0000\u0000\u0000\u0000E\b\u0000(\rì@\u0000\u0001\u0006zÂ,.Ð\u000EÅ*\u0014§(è|P\u0010vmG\u0000\u0000¸ùBl"}
{"bin_timestamp":1623796800,"initial_packet_len":86,"target_ip":2868824378,"target_protocol":6,"attacker_slash16_cnt":171,"attack_port_cnt":593,"target_port_cnt":597,"packet_cnt":599,"icmp_mismatches":0,"byte_cnt":41776,"max_ppm_interval":110,"start_time_sec":1623796672,"start_time_usec":658198,"latest_time_sec":1623797098,"latest_time_usec":681133,"first_attack_port":64110,"first_target_port":29995,"maxmind_continent":"SA","maxmind_country":"BR","initial_packet":"<ýþ\u0019Ø\u0000\u0000Þûº\u0006Ç\b\u0000EÈ\u0000Dÿ¢\u0000\u00001\u0001\u0002ѳ0ðA,¦¶e\u000B\u0000L_\u0000\u0000\u0000\u0000E\b\u0000(V@\u0000\u0001\u0006Ëå,¦¶eªþÉ:únu+'ü\u000EÎI¸ïÖP\u0011ÿÿx\u0000\u0000½2cC"}
Deprecated DoS Data formats
These formats are documented here for users that need to access RSDoS data that was collected using the older corsaro2 software, which has not yet been converted to the latest Avro format.
Corsaro 2 (.csv format)
The data are stored in the Swift container named data-telescope-meta-rsdos-daily.
To list the data files:
limbo@username:~$ swift list data-telescope-meta-rsdos-daily | head
Example
limbo@username:~$ swift list data-telescope-meta-rsdos-daily -d /
year=2008/
year=2009/
year=2010/
year=2011/
year=2012/
year=2013/
year=2014/
year=2015/
year=2016/
year=2017/
year=2018/
year=2019/
year=2020/
To view data:
limbo@username:~$ wandiocat swift://data-telescope-meta-rsdos-daily/year=2016/month=01/day=01/ucsd-nt.rsdos-daily-attacks.2016-01-01.ts=1451606400.csv.gz | head
target_ip,nr_attacker_ips,nr_attacker_ports,nr_target_ports,nr_packets,nr_bytes,max_ppm,start_posix_time,end_posix_time,asn,country-code,continent-code
0.33.69.0,34,1,1,373,22380,34,1451667609,1451670895,,??,??
1.1.1.1,1,12,1,7,280,99,1451636930,1451637056,15169,AU,OC
1.170.208.200,759,754,8,5196,1309558,50,1451606406,1451621421,3462,TW,AS
The example output:
| target_ip | nr_attacker_ips | nr_attacker_ports | nr_target_ports | nr_packets | nr_bytes | max_ppm | start_posix_time | end_posix_time | asn | country-code | continent-code |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.33.69.0 | 34 | 1 | 1 | 373 | 22380 | 34 | 1451667609 | 1451670895 | ?? | ?? | |
| 1.1.1.1 | 1 | 12 | 1 | 7 | 280 | 99 | 1451636930 | 1451637056 | 15169 | AU | OC |
| 1.170.208.200 | 759 | 754 | 8 | 5196 | 1309558 | 50 | 1451606406 | 1451621421 | 3462 | TW | AS |
Data Properties
For each day the dataset is a single compressed CSV file of attack vector. Each attack vector is uniquely identified by the target IP address and the attack start timestamp and contains the following fields:
| Name of Column | Definition |
|---|---|
| target_ip | The IP address of the attack victim (target_ip) |
| nr_attacker_ips | The number of distinct attacker IPs in the attack |
| nr_attacker_ports | The number of distinct attacker ports |
| nr_target_ports | The number of distinct target ports |
| nr_packets | The cumulative total number of packets observed in the attack |
| nr_bytes | The cumulative total number of bytes seen for the attack |
| max_ppm | The maximum packet rate (of backscatter packets) seen in the attack, as a moving average per minute |
| start_posix_time | The timestamp of the first observed packet of the attack |
| end_posix_time | The timestamp of the last observed packet of the attack |
| asn | The autonomous system number of target_ip at the time of the attack |
| country-code | Country geolocation of target_ip, at the time of the attack |
| continent-code | Continent geolocation of target_ip, at the time of the attack |
Corsaro 2 (subset of .csv format)
This dataset is a subset of the RSDoS Attack Metadata. It contains a single compressed CSV files of the attack vectors from March 1, 2015 to February 28, 2017. This dataset was published in Millions of Targets Under Attack: a Macroscopic Characterization of the DoS Ecosystem (2017) by Jonker et al. and is available to non-STARDUST users as well as STARDUST users.
The dataset is not on swift but can be accessed by academic researchers and US government agencies once this form is filled out and submitted.
Note that it may take between two and five business days to process your request.