What is CAIDA?

What is the purpose of CAIDA? What are its goals?

In what collaborations is CAIDA engaged?

What tools is CAIDA developing?

Is CAIDA focused more on tool development or analysis?

What analysis efforts are current/recent/interesting?

Who funds CAIDA?

What is CAIDA's Program Plan?

Who constitutes CAIDA's participants?

How can organizations join CAIDA?

What 'services' are provided to organizations participating in CAIDA?

How does CAIDA collect and take stewardship over traffic data?

How is CAIDA data/analysis disseminated to researchers and to the general public?

Are there other outreach efforts of CAIDA?

What is the role of NLANR?

What overhead does UCSD assess on CAIDA's service agreements?

 

• What is CAIDA?

CAIDA stands for Cooperative Association for Internet Data Analysis. It's a collaborative undertaking among organizations with a strong interest in keeping primary Internet capacity and usage efficiency in line with ever-increasing demand. Participants come from the commercial, government, and research sectors. CAIDA's members use this organization as a focal point for promoting greater cooperation in the engineering and maintenance of a robust, scalable global Internet infrastructure. CAIDA provides the world with a neutral framework to support cooperative technical endeavors that have the potential to be critical in meeting the demands of an exponentially growing system of networks. CAIDA's home is at the San Diego Supercomputing Center (SDSC), an extension of the University of California at San Diego (UCSD) where it was created in 1997 by Dr. kc Claffy and Tracie Monk.

 

• What is the purpose of CAIDA? What are its goals?

CAIDA seeks to address engineering concerns relating to topics that:

• are macro-level, cross-ISP in nature, e.g., infrastructure level measurements and analyses
• evolve technologies to scale up the performance of the Internet, e.g., web caching, multicast
• promote enhanced understanding of traffic trends affecting individual and global Internet infrastructures, e.g., characterization of traffic behavior using active and passive measurements and the analysis of routing data
• improve ISPs' ability to manage their networks through improved traffic analysis and visualization tools

Insofar as possible, CAIDA serves as a facilitator for collaborations in these topic areas, and provides a framework for commercial and research organizations to jointly address areas of common interest. Tools and methodologies developed by CAIDA are intended to have broad applicability within the industry.

 

• In what collaborations is CAIDA engaged?

CAIDA is founded on principles of cooperation and collaboration. Key collaborations currently include:

• NGI/DARPA - In July 1998, the Defense Advanced Research Project Agency (DARPA) awarded UCSD/CAIDA a $2.4 million award for a Next Generation Internet (NGI) initiative, see http://www.caida.org/funding/ngi1998/. This effort focuses on monitoring, depicting, and predicting traffic behavior over production and experimental networks. The NGI project is developing and deploying tools to better engineer and operate networks and to identify traffic anomalies in "real time". Components of this project include: (1) developing tools to automate the discovery and visualization of Internet topology and peering relationships (through application of CAIDA's skitter tool); (2) monitoring and analyzing Internet traffic behavior on high speed links (through the development of OC12 and OC48 monitors); (3) detecting and mitigating threats (through security application of the Coral monitors); and (4) providing for storage and analysis of massive volumes of traffic data. The project leverages existing NSF-supported work, including efforts to evolve and deploy Coral monitors, skitter performance tools, and analysis/visualization techniques.




• SD-NAP - In April 1998, CAIDA announced the formation of a Network Access Point. SD-NAP is hosted by CAIDA at the San Diego Supercomputer Center on the campus of UCSD, with support by both SDSC and the Packet Clearing House (PCH). The primary purpose of the SD-NAP is to facilitate efficient interconnection of IP transit networks within and to the San Diego Local Access and Transport Area, (California LATA 6). A secondary purpose of the SD-NAP is to provide a platform for traffic analysis by CAIDA researchers with the goal of promoting a robust, scalable global Internet infrastructure. Goals and terms of participation, as well as a list of current participants, is available at http://www.caida.org/projects/sdnap/




• one-way performance measurements - In April 1998, Waikato University's Stephen Donnelly visited CAIDA to install a Global Positioning System (GPS) antenna for active one-way IP delay measurements between SDSC and New Zealand.




• cflowd - cflowd is software used to analyze and store flow export data from Cisco's NetFlow functionality. Providers use cflowd in production to analyze flow data from their Cisco routers. CAIDA is working with the community (Cisco and other vendors are moving to implement similar functionality) to continue enhancements to this code. The latest version release is available at http://www.caida.org/tools/measurement/cflowd/.




• Internet Engineering Curriculum (IEC) Repository - CAIDA initiated the Internet Engineering Curriculum Repository project in 1998 under the leadership of Dr. Evi Nemeth and Dr. Claffy. This collaborative project involves the coordination and operation of a Web-based 'living repository' for Internet engineering teaching materials: http://www.caida.org/projects/iec/




• Equipment Loans/Donations - Several organizations participate in CAIDA projects through provision of testing and operational equipment, including Cisco, Compac/Digital, Mae-West, Sun, Cable & Wireless, Juniper Networks, and Zocalo.

 

• What tools is CAIDA developing?

Tools currently under development by CAIDA are described below.

• cflowd - cflowd is an analysis tool used to analyze and store flow export data from Cisco's NetFlow functionality. This analysis package permits data collection and analysis in support of capacity planning, trend analysis, and workload characterization in a network service provider environment. cflowd can also be used for usage tracking for Web hosting, accounting and billing, developing user profiles, and monitoring for security-related investigations. For more information, see http://www.caida.org/tools/measurement/cflowd/.
• CoralReef - CoralReef is a comprehensive software suite developed by CAIDA to analyze data collected by passive Internet traffic monitors. The software presently supports dedicated PC boxes using OC3mon and OC12mon (ATM and POS) cards that collect traffic data in real time. Other monitor interfaces (OC48, gigaether) are under development. CoralReef includes drivers, analysis, web report generation, examples, and capture software. This package is maintained by CAIDA developers with the support and collaboration of the Internet measurement community. CoralReef is the evolutionary successor of the Coral package and supercedes it. Support for OC48 monitors is planned by the end of 2000. For more information, see: http://www.caida.org/tools/measurement/coralreef/.

• Cuttlefish - Cuttlefish produces animated GIFs that reveal the interplay between the diurnal and geographical patterns of displayed data. By showing how the sun's shadow covers the world map, Cuttlefish clearly depicts the time of day at a given geographic region, while moving graphs illustrate the relationship between local time and the visualized events. For more information see: http://www.caida.org/visualization/cuttlefish/

• skitter - skitter is a tool for gathering ICMP TTL (packet time to live) data for mapping Internet peers. It helps engineers analyze paths in the Internet, and its strongest feature is its ability to help visualize BGP routing tables. Engineers can use it to improve topology by obtaining quality metrics from given points on the network. skitter tracks forward IP path (the "hops") from a source to many destinations. Key features of skitter include:
  • lightweight measurement of round trip time and path to thousands of destinations - skitter measures the IP path to a destination in a manner similar to traceroute: it increments the TTL field of successive IP packets sent to a destination and records the router that replies at each TTL until the TTL is sufficient to reach the destination. skitter uses ICMP echo requests as probes, unlike the default of UDP used by traceroute.
  • identification of low-frequency routing changes - Low-frequency persistent routing changes (as well as suggestive evidence of higher-frequency characteristics such as per-packet equal-cost multipath) are visible through skitter measurements. Often a step function in the RTT indicates a change in either forward or reverse path, and a change in the forward path is visible in the skitter data.
  • Depiction of network connectivity - The ability to construct a macroscopic graph of network connectivity (in terms of a directed graph from a source) is a fundamental design goal of skitter. By probing the path to many destinations spread throughout the IPv4 address space, CAIDA can derive a view of a relevant cross-section of the Internet.

  CAIDA is currently monitoring more than 23,000 destination hosts (mostly web servers) from around the globe, from 17 source hosts in the U.S., Europe, and Asia. For more information, see: http://www.caida.org/tools/measurement/skitter/.

  Other utilities in the skitter library include:

  skping - member-sponsored tool for plotting RTT and packet loss, in real-time and over longer history. CAIDA members may get more information at: http://www.caida.org/members/tools/

  sktrace - prototype tool for measuring and visualizing hop-by-hop RTT information. (under development, not yet available)

• arts++ - member-sponsored C++ class library and applications for storing, manipulating and analyzing Internet data, including cflowd and skitter data. For more information, see: http://www.caida.org/tools/utilities/arts/.
• RRDtool - (Round Robin Database tool) is a system to store and display time-series data (e.g., network bandwidth, machine-room temperature, server load average, height of surfing waves on La Jolla Shores). It stores the data in a very compact way, aggregating at stepwise coarser granularity as it archives further back in time, so as to maintain manageable archive size, and it presents useful graphs by processing the data to enforce a certain data density. RRDtool can be used either via simple wrapper scripts (from shell or Perl) or via user-friendly frontends that poll network devices. Tobi Oetiker authored RRDtool during a summer 1999 sabbatical with CAIDA. He continues to support it from his home institution, the Swiss Federal Institute of Technology. For more information, see: http://www.caida.org/tools/utilities/rrdtool/.

CAIDA is also developing topology visualization tools:

• NetGeo -- is a database and collection of Perl scripts used to map IP addresses, domain names and AS (autonomous systems) numbers to geographical locations. CAIDA has developed this system for use in network topology visualization tools. The NetGeo database contains tables for mapping location names (city, state, or country) or US zip codes to latitude/longitude values. These tables are used to map address components found by the NetGeo parser to latitude/longitude. Then the city, state, country, latitude and longitude information is stored with the domain name in the NetGeo database. Phone numbers or email addresses or any other data from the contact section are NOT stored in the NetGeo database. For more information, see: http://www.caida.org/tool/utilities/netgeo/.
• Mapnet - is a tool for visualizing the backbone infrastructure of major Internet Service Providers simultaneously. ISPs may update or correct information that may be invalid or out of date regarding their own infrastructures. The tool's accuracy relies somewhat on the cooperation of providers making current information available. The author of the tool is Bradley Huffaker. See http://www.caida.org/tools/visualization/mapnet.
• Plankton - (no longer supported by CAIDA) Plankton offers a visualization of the NLANR Global Cache Hierarchy topology as seen from the perspective of the NLANR root caches. Bradley Huffaker and Jaeyeon Jung created Plankton, another Java-based tool, in Spring 1998. More information is available at http://www.caida.org/tools/visualization/plankton/
• MantaRay - (no longer supported by CAIDA) Functionality of this tool is now part of Otter, which is described next.
• Otter - Otter is a tool for visualizing nodes and arcs in connectivity data typical to Internet applications. Bradley Huffaker developed Otter in the spring of 1998. Otter currently visualizes a range of data types, from vBNS SNMP data (see http://www.caida.org/tools/visualization/otter/vbns/), to a web tree (see http://www.caida.org/tools/visualization/otter/htmlcrawl/), to relationships among Autonomous Systems (see http://www.caida.org/tools/visualization/otter/as/), to CAIDA's active measurement data (see http://www.caida.org/tools/visualization/otter/). Otter's versatility makes it a critical addition to CAIDA's visualization tools.
• GTrace - is a graphical Java front-end to traceroute that geographically depicts the IP path information between source and destination hosts. GTrace uses a combination of methods to either determine or guess at the physical location of a node in the traceroute path. It is flexible enough to support the addition of new databases, new maps, and new heuristics for mapping IP addresses to physical location. For more information, see: http://www.caida.org/tools/visualization/gtrace/.
• GeoPlot is a lightweight java applet that allows users to create a geographical image of a data set. For more information, see: http://www.caida.org/tools/visualization/geoplot/.
• hypskit (part of skitter toolset) is a tool for visualizing and navigating skitter IP path or BGP routing table data using hyperbolic layout. Daniel McRobb ported Tamara Munzner's 1998 HypView tool and customized it for use with skitter and routing table data. For more information, see: http://www.caida.org/tools/measurement/skitter/viz/hypview/gui.xml.

Tool and Visualization Taxonomies - CAIDA maintains a list of Internet measurement tools and network visualization resources. For more information, see: http://www.caida.org/tools/taxonomy/.

 

• Is CAIDA focused more on tool development or analysis?

CAIDA tends to focus on tools only to the extent that they help ISPs and the Internet community do relevant measurements and analysis. While CAIDA attempts to balance development and analysis efforts, funding and staff limitations leave less time for analysis and outreach efforts than we often hope. CAIDA's experience in these areas continues to make it a solid investment in core Internet measurement technology advancement research, and CAIDA would like to expand its ability to help participants interpret the information acquired through its tools and methodologies.

 

• What analysis efforts are current/recent/interesting?

Measurements of Internet topology in the Asia-Pacific Region Measurements of the Internet topology in the Asia-Pacific Region are being used to examine changes in routing and traffic patterns. This work has already been key to policy and engineering decision making as government and industry reach to meet global demand for affordable increases in bandwidth and decreased latency. See: http://www.caida.org/publications/papers/2000/asia_paper/

Root DNS Server Placement - Key network administrators are co-locating skitter hosts with the DNS root name servers to gather data to determine future, architecturally strategic locations for new root name servers on the Internet. See: http://www.caida.org/tools/measurement/skitter/index.xml#infra

Traffic Trends and Patterns - Long-term traffic trends and patterns at a U.S. Interexchange Point (AMES eXchange). See: http://www.caida.org/publications/papers/2000/AIX0005/.

Workload Profiling Real-time workload profiling of UCSD's CERFnet link to the commodity Internet. See: http://www.caida.org/dynamic/analysis/workload/sdnap/

Daily update Daily skitter probe measurement summaries from active skitter source hosts. See: http://sk-summary.caida.org/cgi-bin/main.pl

 

• Who funds CAIDA?

Both government and industry participate in CAIDA. Most of CAIDA's support is currently provided by government agencies, notably the National Science Foundation (NSF) and the Defense Advanced Research Project Agency (DARPA). Federal funding will continue, focused on longer-term research and development efforts that are beyond the immediate scope or priority of CAIDA's commercial participants. CAIDA seeks to increase commercial funding as a portion of CAIDA's overall budget through targeting development of tools and analysis efforts of particular relevance to providers.

 

• What is CAIDA's Program Plan?

See: http://www.caida.org/home/about/progplan

 

• Who constitutes CAIDA participants?

As a cooperative organization, CAIDA works with individuals and groups with interests coinciding with CAIDA's goals. Informal collaborations are common with specific individuals from ISPs, vendors, universities and research laboratories. Formal collaborations and services are provided through a fee-based participation program. The formal program includes Affiliate, Member, and Sponsor categories at annual dues rates of $7,500, $25,000, and $100,000 respectively.

 

• How can organizations join CAIDA?

If your organization has experience, technology, or skill to offer CAIDA, please contact us using the information at http://www.caida.org/home/contactinfo/. We welcome suggestions for funded projects or other financial assistance as well.

 

• What 'services' are provided to organizations participating in CAIDA?

CAIDA's primary focus is on the development of measurement, analysis, and visualization tools of benefit to the community and pursuit of initiatives to improve the robustness and scalability of the Internet. CAIDA is committed to the effective deployment and the analysis of data acquired by these tools, but must first concentrate on Internet measurement software research and development. CAIDA hopes to empower its participants to better understand the vast amounts of infrastructure measurement data that is now available.

Participants in CAIDA have access to beta versions of tools. Depending on participation level, participants may directly support or collaborate in the development of new CAIDA tools or initiatives. If organizations are participating in CAIDA measurement/traffic analyses, their data will be subject to the provisions of the Proprietary Data Agreement (see http://www.caida.org/home/legal/proprietary.xml).

 

• How does CAIDA collect and take stewardship over traffic data?

CAIDA's traffic data comes from five sources:

• data directly collected from some participating exchange points and networks - these data are proprietary and are not shared
• data provided under NDA or under provisions of confidentiality - these data are proprietary and are not shared
• passive measurement data collected from High Performance Connection universities and FIX-West by the NLANR project - these data are publicly available at http://moat.nlanr.net/Traces/
• routing data from the University of Oregon's Route Views project (see: http://www.antc.uoregon.edu/route-views) and other routing projects, e.g., Merit's IPMA (see: http://www.merit.edu/ipma/)
• active measurement data associated with skitter - Summary graphs and analysis are publicly available at http://sk-summary.caida.org/cgi-bin/main.pl. Raw skitter data files will be made available under AUP agreement to interested researchers at a future point.

 

• How is CAIDA data/analysis disseminated to researchers and to the general public?

CAIDA tools are intended to be made publicly available. Availability of beta versions of some tools, however, may be limited to participating organizations. Traffic data collected from public sources (or from active measurements) will be made available to researchers and others through the CAIDA web pages. Data derived from private sources, however, will remain the property of those organizations; only the cognizant network or vendor can make confidential data available to third parties. Summaries of CAIDA research and analyses will be made available on the web and through presentations and publications. More detailed reports may be made available to participating organizations. If working groups are established under CAIDA, i.e. a metrics working group or an engineering working group, then the members of those working groups will determine what materials may be shared with third parties.

 

• Are there other outreach efforts of CAIDA?

Several outreach efforts are ongoing as part of the Internet Engineering Curriculum (IEC) Project - This is a dynamic repository of Internet engineering curriculum and training materials designed to facilitate training and advancement in fields associated with Internet engineering. IEC outreach includes:

• IEC Workshops - As mentioned above, IEC sponsors annual workshops (at SDSC) to facilitate technology transfer from the authors of networking materials to the faculty who need to use them in University courses. See: http://www.caida.org/projects/iec/
• Traffic Analysis CD - CAIDA has developed and distributed a CD with traffic traces, routing tables, CoralReef analysis software and recommended lab exercises relevant to teaching traffic analysis skills at the graduate and undergraduate levels.
• Internet Teaching Labs - An IEC project extension is coordinating the deployment of Internet Teaching Laboratories (ITL) at several U.S. Colleges and Universities. CAIDA has received a generous commitment of routers, interface cards and software for these laboratories from Cisco systems.

Other community outreach efforts include:

• SD-NAP - the San Diego Network Access Point (SD-NAP) is a neutral network exchange facility that is intended to provide a location for local data network service providers to exchange Internet traffic. It is hosted by CAIDA at the University of California's San Diego Supercomputer Center (SDSC) and is collaboratively supported by SDSC and CAIDA operational networking staff. See: http://www.caida.org/projects/sdnap/
• ISMA Workshops - CAIDA periodically sponsors Internet Statistics and Metrics Analysis (ISMA) workshops at SDSC focused on various topics of Internet measurement, analysis, and visualization. Proceedings and reports are available from: http://www.caida.org/workshops/isma/

 

• What is the role of NLANR?

CAIDA originated as an initiative of Dr. Claffy while working for the National Laboratory for Applied Network Research (NLANR). NLANR and CAIDA became separate entities in 1997, with NLANR focused on measuring links in the research and education (e.g., universities involved in High Performace Computing) community and CAIDA focused on measuring the commodity Internet. NLANR had three components distributed among the San Diego Supercomputer Center (NLANR's Measurement and Operations Analysis Team, MOAT); the National Center for Supercomputing Applications (NLANR's Distributed Applications Support Team, DAST); and the Pittsburgh Supercomputer Center (NLANR's National Center for Network Engineering, NCNE).

The National Laboratory for Applied Network Research (NLANR) Project has officially ended. The funding for the NLANR project expired June 30, 2006 and the National Science Foundation has no plans to continue support. Starting July 2006, CAIDA took over operational stewardship for all NLANR machines and data.

If sites have questions or comments on these changes, or input regarding future measurement experiments, please send them to nlanr-info@caida.org.

 

• What overhead does UCSD assess on CAIDA's service agreements?

CAIDA places significant priority on low overhead costs for funds derived through CAIDA corporate participation. An effective overhead rate of 14% is being applied to service agreements with participating organizations, versus the university's standard overhead of 51.5% on research agreements. Based on discussions with private sector collaborators, this reduced overhead is an important assurance that the majority of commercial participants' funds will apply directly to CAIDA initiatives.

  

---

for more information:   info @ caida.org