skitter Destination Lists - ARCHIVE

CAIDA started the Macroscopic Topology project that uses tool skitter in 1998. Over the years we have been tracking global IP level connectivity by sending probe packets from a set of source monitors to hundreds of thousands of destinations stratifying the current IPv4 address space as well as the Earth.

The destination lists used for various CAIDA Macroscopic Topology studies in the past are listed here.

Name Date Size

Web Servers July 1998 - August 2002 Decreased from ??? to 15.5K

This was the first destination list compiled by CAIDA. Its goal was to provide a representative, world-wide coverage of public webservers.

We created the Web Servers list in mid-1998 by collecting IP addresses of web servers from a variety of logfile sources: caches, web server, search engines. We also traversed parts of the IPv4 address space, using in-addr.arpa to get domain names and then adding 'www' to the beginning of those domain names to test for existence of web servers. Our first skitter monitors used the resulting list to probe thousands of web servers across the world. A year later, we augmented this list with destinations in the Asia-Pacific region.

Intermediate August 2000 - June 2001 Decreased from ??? to 31.5K

The goal of this list was to provide a list of "intermediate" (that is, routers, rather than end user hosts) IP addresses. Studying long term trends in the Web Servers list, we discovered a somewhat surprising result: destinations in our database were becoming unreachable by skitter probe packets at the rate of about 2-3% per month. We started investigating sources of this phenomenon. One avenue of exploration was to find out whether non-leaf IP addresses (i.e., those that are not on last hop of a path) exhibit a similar extinction rate. This Intermediate list resulted, composed of intermediate IP addresses seen in skitter traces from the Tokyo [apan-jp.skitter.caida.org] and Washington, DC [iad.skitter.caida.org] hosts running the Web Servers list on the 20th and 21st of June 2000.

After monitoring this list for 10 months on two skitter montors we found that its extinction rate of 3.0% per months was about the same as the Web Servers list extinction rate of 2.7% per months.

Small July 1998 - February 2001,
and July 2001 - March 2002 <2K

The goal of this list was to provide a subset of the Web Servers list in order to support finer-grain path and performance analysis. When monitoring this list, each host on it was probed 10-11 times per hour.

DNS Clients (old versions) 1st ver: August 1999 - March 2002,
2nd ver: March 2002 - December 2002 58.3K
Decreased from 143K to 108K

The goal of this list was to provide a list of DNS clients that made requests to one of the 13 DNS root nameservers in order to study the connectivity and performance of these root servers.

To create the DNS Clients, we collected IP addresses seen in passive data obtained from a number of root servers, and selected one IP address per routable prefix. If many IP addresses in the same prefix were found, we used different weighting criteria in different versions of this list. Also, about 20% of destinations in the 1st version of this list were added from other sources in order to maximize the coverage of routable prefix space. The 2nd version inherited replying destinations from the 1st version.

IPv4 (old versions) 1st ver: July 2000 - July 2001,
2nd ver: June 2001 - March 2002,
3rd ver: March 2002 - February 2003 313K
567.8K
Subsets: 825K, 330K, 133K

The goal of this list was to find one destination per each reachable /24 segment (256 addresses) of the Internet IPv4 address space in order to collect the best possible comprehensive macroscopic topology data.

We have used a wide range of methods to collect as many IP addresses as possible: tcpdumps from the UCSD-CERF link, hostnames collected from web search engines, IP addresses seen in skitter traces, various log files, etc. From the initial pool of IP addresses we selected those addresses that were on separate /24 segments and replied to our ICMP probes. Note that there are over 16 million potential /24 segments in the IPv4 address space, and about 4 million of them are currently in routable address space. Thus, even for our largest IPv4 lists the coverage is still far from complete.

This project is ongoing. Currently used versions of the IPv4 list are described here.

Name	Date	Size
Web Servers	July 1998 - August 2002	Decreased from ??? to 15.5K
This was the first destination list compiled by CAIDA. Its goal was to provide a representative, world-wide coverage of public webservers. We created the Web Servers list in mid-1998 by collecting IP addresses of web servers from a variety of logfile sources: caches, web server, search engines. We also traversed parts of the IPv4 address space, using in-addr.arpa to get domain names and then adding 'www' to the beginning of those domain names to test for existence of web servers. Our first skitter monitors used the resulting list to probe thousands of web servers across the world. A year later, we augmented this list with destinations in the Asia-Pacific region.

Intermediate	August 2000 - June 2001	Decreased from ??? to 31.5K
The goal of this list was to provide a list of "intermediate" (that is, routers, rather than end user hosts) IP addresses. Studying long term trends in the Web Servers list, we discovered a somewhat surprising result: destinations in our database were becoming unreachable by skitter probe packets at the rate of about 2-3% per month. We started investigating sources of this phenomenon. One avenue of exploration was to find out whether non-leaf IP addresses (i.e., those that are not on last hop of a path) exhibit a similar extinction rate. This Intermediate list resulted, composed of intermediate IP addresses seen in skitter traces from the Tokyo [apan-jp.skitter.caida.org] and Washington, DC [iad.skitter.caida.org] hosts running the Web Servers list on the 20th and 21st of June 2000. After monitoring this list for 10 months on two skitter montors we found that its extinction rate of 3.0% per months was about the same as the Web Servers list extinction rate of 2.7% per months.

Small	July 1998 - February 2001, and July 2001 - March 2002	<2K
The goal of this list was to provide a subset of the Web Servers list in order to support finer-grain path and performance analysis. When monitoring this list, each host on it was probed 10-11 times per hour.

DNS Clients (old versions)	1st ver: August 1999 - March 2002, 2nd ver: March 2002 - December 2002	58.3K Decreased from 143K to 108K
The goal of this list was to provide a list of DNS clients that made requests to one of the 13 DNS root nameservers in order to study the connectivity and performance of these root servers. To create the DNS Clients, we collected IP addresses seen in passive data obtained from a number of root servers, and selected one IP address per routable prefix. If many IP addresses in the same prefix were found, we used different weighting criteria in different versions of this list. Also, about 20% of destinations in the 1st version of this list were added from other sources in order to maximize the coverage of routable prefix space. The 2nd version inherited replying destinations from the 1st version.

IPv4 (old versions)	1st ver: July 2000 - July 2001, 2nd ver: June 2001 - March 2002, 3rd ver: March 2002 - February 2003	313K 567.8K Subsets: 825K, 330K, 133K
The goal of this list was to find one destination per each reachable /24 segment (256 addresses) of the Internet IPv4 address space in order to collect the best possible comprehensive macroscopic topology data. We have used a wide range of methods to collect as many IP addresses as possible: tcpdumps from the UCSD-CERF link, hostnames collected from web search engines, IP addresses seen in skitter traces, various log files, etc. From the initial pool of IP addresses we selected those addresses that were on separate /24 segments and replied to our ICMP probes. Note that there are over 16 million potential /24 segments in the IPv4 address space, and about 4 million of them are currently in routable address space. Thus, even for our largest IPv4 lists the coverage is still far from complete. This project is ongoing. Currently used versions of the IPv4 list are described here.