The IPv4 Routed /24 Topology Dataset

| Data Sources: Realtime Monitors Passive Active Other External |

The IPv4 Routed /24 Topology Dataset

This dataset contains information useful for studying the topology of the Internet. Data is collected by a globally distributed set of Ark monitors. The monitors use team-probing to distribute the work of probing the destinations among the available monitors.

We collect data by sending scamper probes continuously to destination IP addresses. Destinations are selected randomly from each routed IPv4 /24 prefix on the Internet such that a random address in each prefix is probed approximately every 48 hours (one probing cycle). Because team-probing distributes the probing work across all monitors, a single destination /24 will be probed by only one monitor in each probing cycle. The current list of routed IPv4 prefixes was created using RouteViews BGP tables from July 3, 2013. Rather than having a static list of IP addresses to probe, we dynamically pick a new random address in each /24 prefix for every new cycle of probing. The current prefix list includes approximately 10.13 million prefixes (with 6.5 million for data collected before November 2007).

Scamper:

Measures Forward IPv4 Paths

scamper records an IPv4 address seen at each hop from a source to a destination by incrementing the "time to live" (TTL) of each IPv4 packet header, and recording replies from each router leading to the destination host.

Measures Round Trip Times (RTT)

scamper collects round trip time measured to each intermediate router as well as to the destination host.

In the current configuration, scamper probes with ICMP packets, using the Paris traceroute technique (ICMP-paris) to improve measurement integrity across load-balanced links. Data prior to November 2007 used an alternate UDP traceroute method. Data collected for each path probed includes:

RTTs, including both intermediate hops and the destination

IPID, TOS, and TTL, and size fields of response packets

IP length, TTL, and TOS fields of the probe packet that reached each hop (extracted from the response packet)

The ICMP type and code of responses

Scamper also is able to collect Path MTU information, but current measurements do not include that information. A sample binary warts file is available.

Data has been collected continuously since September 13, 2007, and is made available in hour-duration files for the most recent ten days, as well as a historical archive of 24-hour-duration files for the duration of IPv4 Routed /24 Topology project.

Caveats that apply to this dataset:

The IPv4 Routed /24 Topology Dataset uses a dynamic destination list. Measurements to consistent IPv4 addresses are not available in this dataset.

Because team-probing distributes measurements across many monitors, the randomly selected IP addresses in a given routed prefix are not probed by the same set of monitors consistently over time.

Reading Topology Data

You can analyze this data (available in the warts format) with the sc_analysis_dump tool included in the scamper distribution package. The sc_analysis_dump tool prints out information about each trace in an easy-to-parse textual format (one trace per line). You would typically write a perl script to analyze the output of sc_analysis_dump.

Another tool you may want to consider is the warts-dump tool, which is also included in the scamper distribution. The output of warts-dump is somewhat less easy to parse, but warts-dump prints out practically all information contained in a warts file.

Finally, you can write your analysis scripts in the Ruby language using rb-wartslib, an easy-to-use Ruby binding to the warts I/O library.

Data availability

Data older then approximately two years are available as a public dataset. The most recent two years are subject to restricted access.

Acceptable Use Agreement for the restricted data

Access to these data is subject to the terms of the following CAIDA Acceptable Use Agreement (printable version in PDF format)
CAIDA ACCEPTABLE USE AGREEMENT (ver 071814) The following terms comprise the Acceptable Use Policy (AUP) and Data License Agreement (collectively, the "AGREEMENT", or, "Acceptable Use Agreement (AUA)" for all datasets made available to You by the Center for Applied Internet Data Analysis (CAIDA), a research unit at the University of California San Diego (UCSD) and governed by The Regents of the University of California. Certain datasets may have additional Supplemental provisions. References to this AGREEMENT shall include any and all relevant Supplemental provisions. CAIDA has the authority and reserves the right, in its sole discretion, to refuse requests for dataset(s) or discontinue further access and use to anyone. If You feel Your request is inappropriately denied please contact CAIDA by sending a message to data-info@caida.org. In consideration for requesting and receiving access to CAIDA dataset(s), You acknowledge that You understand and agree to be bound by the terms and conditions of this AGREEMENT. Any violation of this AGREEMENT may result in the immediate suspension or termination of this AGREEMENT and/or other action entitled by law such as injunctive or equitable relief. You are individually liable and responsible for compliance with this AGREEMENT. This AGREEMENT is legally binding under the laws of the State of California, United States. You may terminate this Agreement by contacting CAIDA in writing and receiving acknowledgement of such request. 1. LICENSE CAIDA's authorization to access the data grants You a limited, non-exclusive, non-transferable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this AGREEMENT solely for the purpose of non-profit research, non-profit education, commercial internal testing and evaluation of the data, or for government purposes by or on behalf of the U.S. Government. No license is granted for any other purpose and there are no implied licenses in this Agreement. This Agreement shall become effective as of the date of approval by CAIDA and shall remain in force for a period of one year, unless terminated earlier or amended in writing. CAIDA shall have the right to use any of Your feedback received during the license period solely for its non-profit educational and/or research purposes. 2. GENERAL CONDITIONS (i) You will not impersonate any individual or entity, misrepresent any affiliation with another person, entity or association, use false information, or otherwise conceal Your identity from CAIDA at any time for any purpose. (ii) You consent that CAIDA can make public or otherwise disclose Your name as the registered requestor, the name of Your Affiliated Institution, the name of the dataset(s) that CAIDA has made available to You under this AUA, and the brief description of the type of research being undertaken that You provided to CAIDA. (iii) You will abide by any and all modifications. If a modification occurs, it will be explicitly communicated to You via Your registered email address and shall become effective fifteen (15) days after the transmission of such notification. Your continued access to or use of the data after such time shall indicate Your assent to any and all modification(s). If You do not agree to comply with the modification(s), You agree that You will: (a) inform CAIDA immediately, at at which point Your access to the data will be terminated and no longer authorized; and, (b) no longer access or use this data. 3. USE RESTRICTIONS If You have any concerns or questions about these restrictions, You are encouraged to contact CAIDA management via email to data-info@caida.org. To obtain an exemption from any of these restrictions, You will need a written authorization from CAIDA management. (i) While using non-anonymized data set(s), You will respect the privacy of persons that may be identified in the data. For any publication or other disclosure, You will anonymize or de-identify personally-identifiable information, IP addresses, and other data identified in Supplemental provisions (if any) by using commonly accepted techniques such as one of the methods recommended by CAIDA (http://www.caida.org/projects/predict/anonymization/). (ii) While using anonymized data set(s), You will not attempt to reverse engineer, decrypt, de-anonymize, derive or otherwise re-identify anonymized information. (iii) You will not distribute, disclose, transfer or otherwise make available the dataset(s) to any person other than those employed by your institute who are assisting or collaborating with You using the dataset(s). Other entities with whom You are collaborating in research using the dataset(s) must request access to the dataset(s) separately and directly from CAIDA. (iv) All conditions, restrictions and obligations attached to this data shall accompany any and all subsequent uses and disclosures of this dataset by You. Therefore, You are personally and fully responsible for communicating this AUA and ensuring its compliance as to any and all users described above to whom You make the data available. 4. USE OBLIGATIONS (i) If You create a publication (including web pages, papers published by a third party, and publicly available presentations) using data from this dataset, You must provide CAIDA with a copy of (or a link to) the publication and You must cite the data as follows: The CAIDA UCSD [DataSet Name] - [dates used], http://www.caida.org/data/[dataset-URL] (ii) At the end of the research, or semi-annually (whichever is sooner), You will report a summary of the research and any findings/conclusions to CAIDA. This information is used in reports to our funding agencies. (iii) You agree to expunge any and all copies of the received Dataset(s) upon completion or termination of stated research and/or termination of data access or use. Completion of stated research shall allow for a reasonable period of time that You may need to retain the dataset(s) in order to satisfy scientific reproducibility obligations. 5. ACCOUNTABILITY AND ENFORCEMENT (i) You agrees to safeguard any and all sensitive data as required, or may be required, by law by using at least the same degree of care that You uses for its own data of a like nature but no less than a reasonable degree of care, to protect the confidentiality of data and/or the privacy of any identifiable person and to prevent its unauthorized disclosure and use. Data is confidential if it is marked as such, if by its nature or content is reasonably distinguishable as confidential, or if You have reasonable cause to believe that its disclosure to a third party would cause harm or damage. Data is not confidential, and therefore not sensitive if: (a) You already knew the data before it was disclosed to You by CAIDA; (b) You gain subsequent knowledge of the data by either lawfully obtaining it from another source under no obligation of confidentiality, or You develop it independently; (c) the data is or becomes generally available to the public through no wrongful act of You or any other party; (e) it is required to be disclosed under applicable law, regulation or court order provided You notify CAIDA prior to making such a disclosure so that CAIDA may take appropriate action. (ii) You will notify CAIDA immediately of all relevant details if: (a) confidentiality or privacy is compromised; or (b) You receive any legal, investigatory, or other government demand to reverse engineer, decrypt, de-anonymize or otherwise disclose anonymized or confidential data. (iii) If requested, You will provide CAIDA evidence of compliance with this AGREEMENT, such as a written description of Your data protection plan or a written affirmation that You have disposed of any and all copies of the received dataset from all systems reasonably known to you. (iv) DISCLAIMER OFWARRANTIES. CAIDA USES ITS BEST EFFORTS TO PROVIDE DATA IN ACCORDANCE WITH ETHICAL PRINCIPLES AND SCIENTIFIC INTEGRITY. HOWEVER, THE DATA PROVIDED HEREIN IS ON AN "AS IS" BASIS. NEITHER CAIDA, ITS RESEARCHERS, RESEARCH PARTNERS, LICENSORS, AND DATA PROVIDERS, NOR THE UNIVERSITY OF CALIFORNIA AND ITS TRUSTEES, OFFICERS, EMPLOYEES, AND AGENTS MAKE ANY WARRANTY, EITHER IMPLIED OR EXPRESS, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, INCLUDING, BUT NOT LIMITED TO, THE ACCURACY, TIMELINESS, COMPLETENESS, RELIABILITY, OR AVAILABILITY OF CAIDA DATA, APPLICATIONS, OR SERVICES ACCESSIBLE THROUGH OR MADE AVAILABLE BY CAIDA. TO THE EXTENT ALLOWED BY LAW, YOU WILL INDEMNIFY, DEFEND AND HOLD HARMLESS CAIDA AND THE UNIVERSITY OF CALIFORNIA FROMALL LIABILITIES, CLAIMS, DEMANDS, COSTS, JUDGMENTS, DAMAGES, LOSSES AND EXPENSES ARISING OUT OF OR IN CONNECTION WITH ANY USE OF THE DATA BY YOU. IN NO EVENT SHALL CAIDA AND THE UNIVERSITY OF CALIFORNIA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY INDIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL OR PUNITIVE DAMAGES, ARISING FROM YOUR USE OF THE DATA. 6. INTELLECTUAL PROPERTY This section applies only if You are licensing the data for the purpose of commercial internal testing and evaluating the data. (i) Except as specifically provided in paragraphs (ii), (iii) or (iv) of this Section, all right, title and interest in and to any "Intellectual Property" any invention, discovery, improvement, computer or other program, related documentation and work of authorship that is developed for commercial use by You or Your employer, employees or agents, either individually or jointly with CAIDA, using any information provided by CAIDA under this Agreement, shall be the joint property of UCSD/CAIDA and You, and that shared portion is hereby assigned to UCSD/CAIDA, solely for its non-profit educational and/or research purposes. As between CAIDA and You, You retain exclusive and all right, title and interest in and to "Your Intellectual Property" the information originating with or provided by You. (ii) You obtain an exemption from the default Intellectual Property provisions in Section (i) for specific developments via a written authorization from CAIDA management; (iii) You obtain a full Commercial License from the UC San Diego Technology Transfer Office that grants alternative rights in the Intellectual Property; or (iv) You make the Intellectual Property freely available and without restrictions to the public, provided such disclosure complies with obligations regarding sensitive information described in Section 5(i). 6.1 UCSD COPYRIGHT Permission to use, print, copy, and modify any copyrightable part of this UCSD CAIDA Data for educational, research and non-profit purposes, as set forth in the Researcher MOA without fee, and without a written agreement is hereby granted, provided that this paragraph and the following copyright notice and paragraphs appear in all copies: Copyright 2006-2014 The Regents of the University of California. All Rights Reserved. If You desire to use or otherwise incorporate any copyrightable part of this data for commercial purposes, You should contact the UC San Diego Technology Transfer Office, (858) 534-5815, fax: (858) 534-7345.

Acceptable Use Agreement for the public data

Access to these data is subject to the terms of the following CAIDA Acceptable Use Agreement (printable version in PDF format)
CAIDA ACCEPTABLE USE AGREEMENT for PUBLICLY ACCESSIBLE DATASETS The following terms comprise the Acceptable Use Policy and Data License Agreement for all publicly accessible datasets (the "Public Agreement") made available by the Center for Applied Internet Data Analysis (CAIDA), a research unit at the University of California San Diego (UCSD) and governed by The Regents of the University of California. LICENSE CAIDA's authorization to access the data grants You a limited, non-exclusive, non-transferable, non-assignable, and terminable license to copy, modify, and use the data in accordance with this Public Agreement. No license is granted for any other purpose and there are no implied licenses in this Agreement. Nothing in this License is intended to limit any rights You may have arising from fair use or due to other limitations on CAIDA's exclusive rights under copyright law or other applicable laws. CAIDA has the authority and reserves the right, in its sole discretion, to discontinue further access and use to anyone who violates this AUA. If You create a publication (including web pages, papers published by a third party, and publicly available presentations) using data from this dataset, You should cite the data as follows: The CAIDA UCSD [DataSet Name] - [dates used], http://www.caida.org/data/[dataset-URL] We encourage You to provide CAIDA with a copy of (or a link to) the publication. We use this information in reports to our funding agencies. DISCLAIMER OF WARRANTIES. CAIDA USES ITS BEST EFFORTS TO PROVIDE DATA IN ACCORDANCE WITH ETHICAL PRINCIPLES AND SCIENTIFIC INTEGRITY. HOWEVER, THE DATA PROVIDED HEREIN IS ON AN "AS IS" BASIS. NEITHER CAIDA, ITS RESEARCHERS, RESEARCH PARTNERS, LICENSORS, AND DATA PROVIDERS, NOR THE UNIVERSITY OF CALIFORNIA AND ITS TRUSTEES, OFFICERS, EMPLOYEES, AND AGENTS MAKE ANYWARRANTY, EITHER IMPLIED OR EXPRESS, OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, INCLUDING, BUT NOT LIMITED TO, THE ACCURACY, TIMELINESS, COMPLETENESS, RELIABILITY, OR AVAILABILITY OF CAIDA DATA, APPLICATIONS, OR SERVICES ACCESSIBLE THROUGH OR MADE AVAILABLE BY CAIDA. LIMITATION OF LIABILITY. TO THE EXTENT ALLOWED BY LAW, IN NO EVENT SHALL CAIDA AND THE UNIVERSITY OF CALIFORNIA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY INDIRECT, CONSEQUENTIAL, INCIDENTAL, SPECIAL OR PUNITIVE DAMAGES, ARISING FROM YOUR USE OF THE DATA. If You have any questions about the data or about this Public Agreement, please email data-info@caida.org.

When referencing this data (as required by the AUA), please use:
The CAIDA UCSD IPv4 Routed /24 Topology Dataset - < dates used >,
http://www.caida.org/data/active/ipv4_routed_24_topology_dataset.xml.
Also, please, report your publication to CAIDA.

Request Data Access

Access the publicly available CAIDA IPv4 Routed /24 Topology Dataset (and other topology data)

Request Access to the restricted CAIDA IPv4 Routed /24 Topology Dataset (and other topology data)

Topology Datasets

Freely Available Datasets

The Ark IPv4 Routed /24 Topology Dataset (data older than two years only)

The Ark IPv4 Routed /24 DNS Names Dataset (data older than two years only)

Ark Internet Topology Data Kits (ITDK) (data older than two years only)

The Ark IPv6 Topology Dataset

The Ark IPv6 DNS Names Dataset

The IPv6 Routed /48 Topology Dataset

IPv4 Routed /24 AS Links (September 2007 - ongoing)

IPv6 AS Links (December 2008 - ongoing)

AS Rank

AS Relationships

Skitter Macroscopic Topology Data

Skitter Internet Topology Data Kits (ITDK) - April 2002 and April/May 2003

Skitter AS Links (January 2000 - February 2008)

Skitter Router Adjacencies

AS Taxonomy

PAM 2010 "Improving AS Annotations" Supplement

Restricted Access Datasets

The Ark IPv4 Routed /24 Topology Dataset (incl. most recent two years)

The Ark IPv4 Routed /24 DNS Names Dataset (incl. most recent two years)

The Ark IPv4 Prefix-Probing Dataset (incl. most recent two years)

Ark Internet Topology Data Kits (ITDK) (incl. most recent two years)

Complete Routed-Space DNS Lookups

References

For more information on CAIDA topology measurements, see:

Internet Topology Data Comparison

The Archipelago Project (Ark)

Macroscopic Topology Measurements

For more information on topology measurements in general see:

University of Oregon Route Views Project

Paris traceroute