cooperative association for Internet data analysis

cooperative association for Internet data analysis
(CAIDA)

A proposal submitted to
the National Science Foundation


Summary of proposal

The Cooperative Association for Internet Data Analysis (CAIDA) seeks to promote greater industry cooperation in architecting and managing the Global Internet Infrastructure. It will address global engineering concerns that are highly dependent upon cross-ISP coordination, particularly those requiring measurement of Internet metrics. It will also address ISPs' emerging need for technical mechanisms to facilitate service guarantees and financial settlements between providers. Specifically, the CAIDA will endeavor to:
  1. identify, develop and deploy measurement tools across the Internet;
  2. work with commercial providers to provide them with a neutral, confidential vehicle for data sharing and analysis;
  3. provide networking researchers and the general Internet community with current data on Internet traffic flow patterns;
  4. assist in the introduction / deployment of emerging internet tools and technologies such as multicast, IPv6, web caching, bandwidth reservation protocols, etc.; and
  5. enhance communications between commercial Internet service providers and the broader Internet communities.
The goal is to have both government and industry participate in CAIDA's creation. The initial vision of CAIDA is as a supporting framework for a set of active tasks to be defined in real time in conjunction with CAIDA members. There will be three modes through which tasks will be defined: proposed by CAIDA researchers themselves to an individual or set of CAIDA sponsoring members; jointly proposed by CAIDA researchers and CAIDA sponsoring members; or solicited by a single or set of sponsoring members to CAIDA.

We are proposing that NSF help to seed the effort as a charter CAIDA member. This proposal covers the initial three years of seed support for CAIDA research and development efforts. Complementing an industry-wide effort with government support will promote balance among the needs of the various communities (private, research, government, and users), and facilitate the near term development and deployment of critical measurement technology and techniques.


1.0 Internet Infrastructure Consortium


1.1 Current Situation (https://www.caida.org/catalog/papers/1996_metricsurvey/metricsurvey/ has an NLANR survey of current activities)

As the era of the NSFnet Backbone Service came to a close in April 1995, the community lost the ability to rely on what was the only set of publically available statistics for a large national U.S. backbone. The transition to the new NSFnet program, with commercial operations providing both regional service as well as cross-service provider switching points (NAPs), has virtually eliminated the public availability of statistics and analysis at the national level.

Currently, there is no centralized control over Internet service providers (ISPs). ISPs do not always coordinate their efforts with each other, and quite often are in competition. Nevertheless, the National Information Infrastructure (NII) continues to drive funding into hardware, pipes, and multimedia-capable tools, with very little attention to any kind of underlying infrastructural sanity checks. The market for IP access provision is now estimated at more that $720 million -- upwards of $2 billion if information providers such as AOL and Compuserve are included. (src: Maloff Company's 1996 IAP Marketplace Analysis Report)

Obstacles to the collection and analysis of traffic data on the commercial Internet include political, legal (privacy), logistical, and proprietary considerations. Data acquisition will be further complicated by networks transitioning to high speed switched technologies (e.g., ATM, switched FDDI). In this environment, it may no longer even be technically feasible to access IP layer data in order to do traffic flow profiling, certainly not at switches within commercial ATM clouds. Many newer layer 2 switches, e.g., DEC's gigaswitch, and ATM switches generally, have little if any capability for performing layer 3 statistics collection, or even looking at traffic in the manner allowed on a broadcast medium (e.g., FDDI, Ethernet), where a passive listener can collect statistics without interfering with switching. Statistics collection functionality in newer switches takes resources directly away from forwarding of frames/cells, which may have the effect of driving customers toward switches from competing vendors who sacrifice such functionality in exchange for speed.

Recognizing the implications of these transitions, the National Laboratory for Applied Networking Research (NLANR) took steps in mid 1995 (with funding by NSF and support by the Federal Networking Council) to position measurement equipment at the federally-owned FIX-West exchange point. Results of FIX-West traffic characterization have been presented at numerous venues over the last year, including the Federal Networking Council Advisory Committee meetings, NANOG, and IETF meetings.

NLANR's mission extends to serving `as a vehicle to integrate issues in economics and cultural network penetration with application, architecture and engineering research'. In this capacity, NLANR worked with Bellcore to host an NSF-supported workshop entitled Internet Statistics and Metrics Analysis in February 1996. Workshop participants articulated a need for smoother coordination and information exchange among Internet service providers -- both providers of Internet access and traffic exchange services.

Although most participants at the workshop felt that the Internet development and provider community should seek out measurement infrastructure and sources of statistics in the commercially decentralized Internet, there was definite dissonance as to which measurements would help, and who should have access to them. While a public infrastructure for `end-to-end' measurements could help researchers and end users study the infrastructure, participants concurred that Internet service providers (ISPs) themselves would be the greatest beneficiaries of an enhanced statistics capability. Opinions varied as to the sensitivity of some data and how much could be released publically. Participants agreed that a neutral venue was needed for the publication of such statistics. A neutral forum could also facilitate more comprehensive collaboration, consensus-building, and the development of tools to measure new metrics that are important to stability, service, efficient resource allocation, and more economically viable network usage pricing policies.

NLANR (with KC Claffy, UCSD serving as PI) proposes to establish a neutrally sponsored measurement and analysis team that could eventually serve as a critical component of such a forum. Indeed, this team would provide technical grounding to effectively launch the forum, with a concrete, specific agenda that will also promote longer term service improvements on the Internet, The CAIDA team will concentrate on the needs outlined at the ISMA workshop>: the exchange of statistics that would assist the engineering of a multi-ISP infrastructure. ISPs, who have thus far not devoted priority resources to statistics collection, will find an increasing need for them as network technology, infrastructure, utilization, and cross-section of customer profiles grows.

1.2 Need for an Industry/Research Consortium

In Routing in a Multi-Provider Internet, Y. Rekhter writes

    Despite all the diversity among the providers, Internet-wide IP
    connectivity is realized via Internet-wide distributed routing, which involves multiple providers, and thus implies certain degree of cooperation and coordination. Therefore, there is a need to balance the providers' goals and objectives against the public interest of Internet-wide connectivity and subscribers' choices. Further work is needed to understand how to reach the balance.

Through this proposal, we hope to promote greater collaboration between both the Internet access and exchange providers and Internet users. The proposal sets forth tangible goals of:

  • identifying, developing and deploying measurement tools across the Internet;
  • working with commercial providers to provide them with a neutral, confidential vehicle for data sharing and analysis;
  • providing networking researchers and the general Internet community with current data on Internet traffic flow patterns; and
  • enhancing communications between commercial Internet service providers and the broader Internet communities.

1.3 Role of the Government

Federal agencies have expressed interest in playing a continuing role in the development, testing and deployment of new networking technologies. Indeed, although a less powerful force in the operational Internet than in the past, as a major user with mission-critical Internet requirements, the federal government continues to play a role in facilitating discussions about the Internet architecture and administration.

The Federal Networking Council Advisory Committee (FNCAC) addressed this role in their December 1996 recommendations on Internet Statistics and Metrics. According to the FNCAC, FNC agencies should strive toward:

    • continued funding for research on Internet performance methodologies with a particular emphasis on the development of suites of tools that could be adopted by the Internet industry as standard metrics.
    • deploying these tools in both their production and experimental networks, and strong encouragement that industry should develop and employ such suites of tools.
    • funding studies of interpretation of Internet statistical data and its trends. The results of these studies would be shared with network providers to help them optimize their operations and engineer improvements using new technologies and protocols.
    • encouraging those agencies traditionally involved with the collection and dissemination of industry-wide statistics and data on other national resources to become involved in the area of Internet statistics.
    These recommendations were consistent with their earlier recommendation in May 1996 that the agencies
    • stimulate the formation of an ISP-industry-wide neutral body empowered (by its constituent members) to make measurements and quantify performance.

    Consistent with these sentiments, government agencies are initiating internal efforts to assess the state of the Internet and the implications of increasing congestion upon their ability to achieve agency missions and serve their research constituencies. (See the meeting summaries.)

    Given the level of measurement research and data acquisition/analysis underway by or for federal agency networks, we anticipate forming close, cooperative technical relationships with specific federally sponsored groups, for collaboration on the development of complementary measurement tools and testing and deployment of measurement platforms across the federal R&E network infrastructure. In addition, certain federal agency networks may choose to participate as members of the consortium.

    Similar technical relationships and membership may also be appropriate for non-U.S. government research networks. Research networks in Europe, for example, do not utilize commercial backbones, and may be particularly interested in participating in both the research and data sharing components of the consortium. Within the constraints of U.S. antitrust law, who participates in what capacity will be determined by consortium members.

    1.4 Data/Tools as the Core/Foundation of a Consortium

    Promoting a limited exchange of statistics should have direct payback in achieving ISP goals such as configuring and managing interconnections based on the traffic profiles of constituent peers. We expect that ISPs will only participate in limited data exchange within the closed consortium if a neutral, independent, well-trusted, technically capable team coordinates it and assumes responsibility for collection of select statistics. Through the consortium structure, the CAIDA team would offer services and tools to encrypt sensitive data, process log files, reduce and visualize large data sets, and provide interactive, access-controlled access to customizable reports for consortium members.

    The technical emphases would be on: (1) testing the utility of such a forum as a vehicle to address engineering concerns that would benefit from cross-ISP coordination; and (2) developing and proposing of a set of standard criteria by which consumers could compare the quality of service offered by different ISPs. Associated policy emphases would include establishing a framework which, over time, could be used to promote greater industry cooperation in the architecting and management of the global Internet infrastructure.

    Examples of statistics analyses of immediate relevance to providers include using measurements of: round-trip-time (RTT), e.g., with ping, to assess congestion and other conditions at an infrastructure-wide level; routing behavior (extensions of data and tools from the Routing Arbiter project), to assess status and stability, as well as unusually configured routing and more specific routes in the presence of less specific routes. The consortium could also work with member ISPs to define and engineer multicast and caching topologies that would benefit from leveraging across ISPs, and seek mechanisms to promote data sharing in areas such as matrices of traffic flow among autonomous systems (ASMs) and network outage reporting. Examples of information that might be published include round-trip-time figures collected from a variety of sources.

    We hope to form the consortium through the participation of major ISPs. AT&T, for example, has expressed support for the initiative and interest in being among the founding members. Microsoft, MCI, Nokia, Sprint, Cisco, ANS, and BBN Planet have all also expressed interest in participation. Digital, Ipsilon, ANS and BBN have already contributed human resources to CAIDA activities. Participation of major ISPs and vendors would both distinguish them as industry leaders and set them apart from providers who lack a fundamental commitment to the overall Internet infrastructure. Data collection would be facilitated in one of two ways (depending on the preference of members):

    1. CAIDA, in cooperation with member ISPs, the NAPs and various other exchange points, would collect data using tools such as modified pings and Mathis' Treno over IP networks, and utilize emerging tools as they become available.
    2. Members of the consortium would send data, appropriately encrypted, to the CAIDA team (in its function as the neutral trusted third party). The CAIDA team will distill it and present collated statistics to consortium members, initially via an access-controlled web server.
    The emphasis of all data collection will be on analyzing statistics that help the collective engineering and evolution of the overall Internet environment, and specifically the consortium members. While retaining an otherwise competitive and commercial arena, this forum will allow, at the engineering level, derivation of a set of common service metrics which are critical to ensuring the continued vitality of the Internet.

    On a case-by-case basis, and with proper privacy agreements, the consortium will likely want to allow researchers, within or affiliated with member companies, access to specific data samples for the purpose of developing estimation and prediction methodologies, and vendor engineers similar access for the purpose of improving router operation. While consortium members may ultimately choose to make data publically available, we anticipate that such availability would be gradual, dependent upon members' realization of tangible benefits related to dissemination. In addition, we anticipate that the members may choose to impose strict guidelines and/or penalties for misuse of consortium data. In particular, provisions in the bylaws (or other mechanisms) may be employed to ensure that network sensitive data is not exploited or otherwise misused for competitive advantage by a consortium member.


    2.0 Deliverables and Timeline



    Our project deliverables are in the spirit of many of NCRI's projects that have established research prototypes to seed the infrastructure (e.g., the NSFNET, RA, NAPs, the NLANR caching project). We intend for CAIDA to follow in this line of successful infrastructural prototypes, and provide an initial baseline set of metrics, tools and a user guide to support cooperative analysis in the new environment. The tool set will integrate components of other tools under development in the community, and include extended development on two CAIDA tools: anemone, a X-window based tool for network graph drawing, visualization, and analysis, and fish, an ICMP based tool for end-to-end path performance assessment. Because reasonable interpretation of measurement results is critical to fair assessments and evaluations, we will publish guidelines based on the consensus of representatives from the ISP, network research, and large user communities.

    Many of the project deliverables will be administrative: clarifying and refining concepts of collaboration/cooperation and organizational structure/relationships among ISPs/users, formalizing the technical processes, expanding participation (including internationally), and supplementing/replacing public funding with private sector contributions as CAIDA moves toward a fully independent non-profit organization. We detail the timelines of these deliverables in the following section.

    We will also provide quarterly and annual status reports. Quarterly reports will detail

    1. status of tool development and deployment, including information on the organizations engaged in the efforts and plans for their public and private use
    2. summary of data collection, aggregation and analysis activities of CAIDA and affiliated research institutions
    3. pointers to publications or other materials developed as part of this effort
    The first annual report will focus on the administrative development of CAIDA, including its bylaws, membership, antitrust and privacy agreements, financial status, and other relevant details. The second annual report will focus on the status and accomplishments of CAIDA with respect to facilitating global Internet measurement activities and related Internet engineering/operational collaborations. This report will also include a summary of research goals and other requirements for the final year of the grant. The final report will summarize accomplishments in tool development/deployment; data acquisition and analysis; and technology transfer/cooperation among the participating member companies and affiliated research/academic institutions.

    In addition to these deliverable reports, CAIDA will prepare and distribute informal outreach materials via the web or paper copy. These materials will include specific analysis of Internet traffic conditions or other aspect of Internet operations.

    2.1 Phase I: the building blocks (0 - 12 months)

    a. developing mechanisms (tools/processes) for data acquisition/sharing

    Measurements that can provide providers with data relevant to maintaining and extending their networks are perceived as having a positive return on investment for commercial service providers. With this critical business motivation in mind, we propose to initially target statistics that will support:

    1. network architecture design and resource allocation policies. Accurate data on traffic patterns will allow engineers to design more efficient architectures, and design them more quickly, conserving both labor and resources unnecessarily allocated to parts of the network where they are not needed.
    2. faster problem resolution, saving the time and money now devoted to chasing down problems, e.g., route leakage, link saturation, route flapping.
    3. identify/develop critical networking metrics and tools, including defining the characteristics of an ideal measurement tool that could gather data on both end-to-end performance and workflow characteristics
    4. the development of more effective usage-based economic models, which ISPs are (sometimes reluctantly) admitting will be essential to viable operation.

    b. clarifying concepts of collaboration/cooperation among ISPs/users

    Given the nascent stage of the market and the rapid growth characteristic of the industry, there has been little time or attention to collaboration/cooperation among service providers and between providers and the user community. During the first months of this effort, we will develop and implement a framework for interaction among these groups.

    c. clarifying organizational structure/relationships

    Legal issues are fundamental to the definition of an organizational structure for the consortium. During the initial stages of this effort, UCSD will work directly with the legal departments of the founding members to identify legal and privacy considerations related to the new forum and to draft the appropriate agreements and other documents.

    During this year, CAIDA staff will begin the process of weaning the consortium from the UCSD administrative infrastructure. Alternatives for bookkeeping, legal and other support will be explored in anticipation of becoming an autonomous entity during year 2.

    d. establishing the non-profit organization

    By the second half of year 1, the consortium will be formally chartered as a 501 (c) non-profit organization. This step will include agreement upon the consortium's bylaws, antitrust and privacy agreements, and application. The Board of Directors and corporate officers will also be identified.

    2.2 Phase II: implementing the vision (6 - 24 months)

    a. formalizing technical processes

    Formalizing technical processes will begin concurrent with organizing the consortium. Table 1 lists some possible metrics and an indication of its relevance to Internet settlements and/or workload/performance evaluation. The CAIDA team will coordinate with founding consortium members to set priorities on which metrics to track and which tools to develop/deploy, and how to collect and visualize/present the data. They will then work with the ISPs and other research groups (as appropriate) to implement these priorities.

Internet Metrics & Tools

Table 1: Internet Metrics and Tools
TypeApplicable whereRelevance to Internet SettlementsRelevance to Analysis of Network PerformanceMeasurement Tools
Raw Metrics:
- Access Capacity (bit/sec) CC charge for bit rate; equipment cost depends on bit rate
high
low
a priori
- Connect Time CC charge for connect time
high
low
CC metering
- Total Traffic (bytes) transit traffic settlement between ISPs
high
low
router or access server stats; packet/flow sampling.
- Peak burst (bit/sec sustained for n sec.) ISP/NSP overbooks trunks
moderate
moderate
ANS's cflowd, tcpdump RTFM meters; etc.
Routing:
Announced Routes (#)at exchange points, multihomed connections
high
moderate
RA data and tools, netaxs
- Route Flaps (#)exchange points, multihomed connections
moderate
high
RA tools
- Stability, e.g. route up/downtime, route transitionsexchange points, multihomed connections
low
moderate
RA tools
- Presence of more specific routes with less specific routesexchange points, multihomed connections
moderate
moderate
RA
- Number of reachable destinations (not just IP addresses) covered by a route exchange points, multihomed connections
moderate
moderate
RA
Path Metrics:
- Delay (milliseconds)everywhere
moderate
high
TBD, ping
- Flow Capacity (bits/sec)everywhere
moderate
high
TBD, treno
- Mean Packet Loss Rate (%)everywhere
moderate
high
TBD, ping-based
- Mean RTT (sec)everywhere
moderate
high
TBD, ping-based
- HOP Counts/Congestioneverywhere
low
high
TBD, traceroutes
Other:
- Flow characteristics (protocol profiles, cross-section, traffic matrix, asymmetry) exchange points, multihomed networks
low
moderate
ISP report
- Network outage information (remote host unreachable)individual networks
low
moderate
ISPs reports, IPNmoo
- AS x AS matricesIndividual networks
moderate
low
ISP reports
- Information Sourceconnection of service provider (DNS or RR server); content provider (web server); info replicator (MBONE router & caches)
high
high
router or access server stats; packet sampling, flow meters; etc.
Topology Visualization:
- MBONE Internet Infrastructure
moderate
moderate
TBD
- Information caching hierarchyInternet Infrastructure/individual caches
moderate
moderate
TBD

Notes: CC - common carrier, ISP - internet service provider, NSP - national service provider, TBD - to be determined

Sources: Metrics for Internet Settlements, B. Carpenter (CERN), Internet Draft, May 1996;
A Common Format for the Recording and Interchange of Peering Point Utilization Statistics, K. Claffy (NLANR), D. Siegel (Tucson NAP), and B. Woodcock (PCH), presented at NANOG, May 30, 1996.


Many of the Internet metrics above are inherently problematic, and still require research, in areas such measuring one-way delay, metrics for variance and other statistics of delay distribution (e.g., percentiles), and dealing with asymmetric routing. For example, for a given delay measurement, total delay across a path equals the sum of the delay across the components of the path. It is not clear what analogous statements one can make regarding other delay statistics e.g., mean, variance, percentiles. Vern Paxson's Towards a Framework for Defining Internet Performance Metrics ( compressed postscript here) published in Inet '96, presents a cogent discussion of these issues.

Measuring the throughput or flow capacity of a live connection without detrimental performance impact is difficult. In theory, flow capacity is also amenable to path decomposition, i.e., one can approximate the flow sustainable across a path by the minimum of the flows sustainable across each of the components of the path. In practice, however, buffering characteristics and routing asymmetries impose `friction' in the system, limiting the viability of the formal definitions. We will need to compare several alternatives for empirical metrics, and qualify the degree to which they deviate from the corresponding formal metrics. It will be essential to develop a methodology to estimate sustainable throughput from some baseline flow capacity measurements in conjunction with current delay measurements. NLANR researchers at PSC have developed one possible tool to measure transport layer performance, treno, with a descriptive paper published in Inet '96.

Future data activities may stretch beyond the network layer toward optimizing overall service quality via mechanisms such as information caching and multicast. Visualization is important for making sense of all the data sets described above, and is especially critical for developing and maintaining the efficiency of logically overlaying architectures, such as caching, multicast, mobile, IPsec, and IPv6 tunnel infrastructures. Examples include

b. expanding participation

At this stage, expanding participation will receive high priority, to include additional representatives from four illustrative categories: 1) National Service Providers (NSPs) and National Access Point (NAP) providers; 2) Internet Service Providers (ISPs) and Peering Exchange providers; 3) Special Members, including equipment and software venders; and 4) Research Networks, including Federal and internationally recognized research networks (e.g., networks participating in the Coordinating Committee for Intercontinental Research Networks, CCIRN) . Participation by members will be in terms of both private sector financial support and data acquisition/sharing.

c. developing analytic capability

Initial tool development and data analysis priorities will be set during the first six months. The CAIDA team will collaborate with other principal investigators and research institutions, including participants in the Common Solutions Group (CSG) proposed measurement infrastructure initiative. The contortium will establish guidelines through which researchers (from academia, telecommunications firms, and elsewhere) can submit brief proposals requesting access to CAIDA-specific data sets and will encourage collaborating institutions (e.g., the CSG initiative) to do likewise. Criteria for access will include the benefits of the research to the overall Internet community and the willingness of researchers to abide by the consortium's guidelines with respect to privacy and confidentiality.

Such collaborations will be critical to furthering the state-of-the-art in this sector and ensuring that information sharing among the numerous international entities with interests in this area are kept informed and/or directly involved.

2.3 Phase III: weaning the consortium (12 - 36 months)

a. supplementing/replacing public funding with private sector contributions
While non-government financial contributions are anticipated during the first year, we expect that CAIDA will institute a formal grant structure following the establishment of the non-profit organization. Initial funds may supplement organizational/technical activities, and during the second year will begin to replace government funds.

b. encouraging international participation
Given the global nature of the Internet and the importance of concepts such as resource reservation (e.g., RSVP) and settlements, we anticipate particular interest in the consortium from international service providers and telecommunications firms such as British Telecom, Singapore Telecom, and others.

c. reassessing charter/structure
During the third year of operation, direct involvement by government in the consortium will transition to that of a member. We anticipate that the government may continue to fund R&D related to networking statistics/metrics, but that industry will also take advantage of CAIDA as a resource upon which to draw for studies they need but do not have resources to pursue themselves. The goal is to have both government and industry participate in CAIDA's creation. The initial vision of CAIDA is as a supporting framework for a set of active tasks to be defined in real time in conjunction with CAIDA members. We expect there will be three modes through which tasks will be defined: proposed by CAIDA researchers themselves to an individual or set of CAIDA sponsoring members; jointly proposed by CAIDA researchers and CAIDA sponsoring members; or solicited by a single or set of sponsoring members to CAIDA.


3.0 NLANR's Role

3.1 NLANR's Involvement

NLANR provides the NSF and other Federal agencies active in networking research and development with the unique opportunity to create a distributed national laboratory for applied network research. Integrated into the infrastructure of multiple existing NSF supercomputing centers and Federal R&E networks (most notably NSF's high speed vBNS networking activity), NLANR serves as a framework for a collaborative agenda to evolve national networking. The collaborative framework is specifically designed for its development into an important national asset for the NSF and the U.S. research and education community.

In the statistics arena, NLANR and those associated with it have distinguished themselves as leaders in Internet measurement. At SDSC, Hans-Werner Braun and K.C. Claffy collected and analyzed traffic flows through the FIX-West exchange point for more than a year. Claffy and Bilal Chinoy have worked with the NSF-supported NAPs to make public their traffic data. Claffy, Dave Siegel (RTD) and Bill Woodcock (Zocalo) have developed and deployed common data file formats across Metropolitan Internet exchange points (MIXes). Eric Hoffman (prev. Ipsilon, now with NLANR) is developing a tool for monitoring congestion along Internet paths.

Other NLANR sites are also engaged in measurement activities. Matt Mathis and Jamshid Mahdavi (PSC) are continuing their work on TCP performance issues with Sally Floyd (LBL) and Allyn Romanov (Sun), presenting the SACK Internet Draft at the March 1996 IETF, and publishing a related paper in Sigcomm '96. Both are currently working on implementations, focusing on NetBSD and Digital Unix, which they will test over both the vBNS and commodity network as soon as they are completed.

3.2 Liaison to R&E community: (NIMI, Common Solutions Group, DOE and NASA activities)

While Caida will ultimately be an institution to directly serve the ISP community and its commercial infrastructure, NLANR will provide essential glue, for at least several years, to test and deploy tools on R&E infrastructre before launching them on core commerical infrastructure. Several NLANR investigators are deeply involved in community measurement efforts, making CAIDA support a natural extension of its mission.

NLANR researchers Matt Mathis and Jamshid Mahdavi of PSC, in collaboration with Vern Paxson (LBL), have proposed a National Internet Measurement Infrastructure on which to deploy performance assessment software. They intend to enhance the current NPD software to be more industrial strength, but hardly more intrusive than ntpd, and then initially deploy it on research and federal agency infrastructure. Federal agency networks such as DREN and AAInet have already expressed interest in contributing bootstrapping resources to such a measurement infrastructure. In particular, the DOE has already begun extensive measurements to ascertain the performance their users receive, both domestically and internationally. NASA has already and will continue to provide NLANR with support for placing a measurement machine at FIX-west, which has become one of the few if not only publically available source of workload characteristics on the public Internet.

In the educational domain, the Common Solutions Group have expressed plans to make selected campus infrastructure available for deploying objective performance assessment tools, has also already approached NLANR about tool deployment on their infrastructure.

NLANR would offer or coordinate initial support for probes as desired by these communities, as CAIDA focuses on fostering research relationships with the commercial community.

3.3 NLANR/RA collaboration

Craig Labovitz (Merit), under the auspices of the NSF-funded Routing Arbiter project, offers an extensive collection of data and statistical reports. Data include real-time graphs of network instability, charts and reports of instability and invalid routing announcement sources, and long-term trends. Also measured are packet loss and delay across and between the NAPs. The RA also uses SNMP to track BGP update totals, idle peering sessions, system uptime, free virtual memory and CPU utilization. This collection of data provides insight into end-to-end routing pathologies, provider stability and network topology. NLANR has been working with Craig for several months, and he will be the PI on a CAIDA-adjunct proposal to be submitted from the University of Michigan for routing statistics analysis and tool development. We intend for the University of Michigan activities to gradually become seamlessly integrated with CAIDA.

3.4 CAIDA as an Autonomous Organization

The NLANR project supports the national networking agenda, providing long range engineering, design, and planning support through leveraging the resources of the NSF-supported supercomputing centers and other research institutions, including Harvard University (Scott Bradner), Lawrence Berkeley Labs (Vern Paxson), Kansas University (Gary Minden and Joe Evans), and Bellcore (Padma Krishnaswamy). NLANR's role in the formation of the proposed consortium is consistent with elements of its mission, most notably, those NLANR goals related to:

  • widening access to shared national computing, networking, and laboratory resources
  • speeding the review and dissemination of NLANR research results to the scientific and other scholarly communities in academia, government agencies, and private industry
  • establishing a vehicle for integration of network and application-based objectives, which will help the United States remain a leader in networking technology
Given industry's nascent level of understanding concerning Internet measurements, including measurements associated with emerging ATM infrastructure, NLANR can significantly contribute to the development and implementation of new measurement technologies and techniques. Over time, however, use of these technologies will become more commonplace, and organizational control of CAIDA will transition from UCSD/NLANR to officers of the non-profit organization. At that time, CAIDA's affiliation with the university and with NLANR research staff will be reexamined in light of CAIDA priorities and goals of the member companies.

4.0 Key Events



4.1 Organizing summit - March 1997

We hope to hold an organizing workshop meeting of founding members and other key individuals around March 1997. Specific goals for this meeting include reaching agreement about

  • the purpose and initial membership of the consortium
  • membership and grant structure.
  • plan for development of chartering documentation, including membership and privacy agreements, research agenda, business plan, and budget.
  • a recommended baseline level of data to be provided by industry to the consortium
  • procedures for ensuring data integrity and confidentiality
  • types of data to be made available to research community
  • types of data to be made available to general Internet community

4.2 Follow-on to ISMA - April 1997

NLANR's role will be to assist in the design of the workshop. In her role as as PI of the CAIDA measurement and analysis research team, KC Claffy would encourage participation by other members of the consortium and lead discussions related to consortium requirements with respect to tool development/deployment and data analysis. Another institution might contribute with logistical coordination of this event (with government/private sector funding). The RFC's discussed during the IETF's Internet Provider Performance Metrics (IPPM) working group session in December 1996 would be covered, as would initial priorities identified by CAIDA members.

Goals include:

  • heightening awareness of researchers/service providers with respect to the state-of-the-art in this area.
  • identifying technical constraints related to achieving pre-defined service provider goals for Internet statistics/metrics
  • setting specific research objectives with respect to the development and deployment of specific tools
  • encouraging cooperation among the various participating Internet communities
4.3 Other Meetings

An initial meeting of the Board of Directors is tentatively scheduled for mid-1997. At this meeting, the Advisory Members will review progress related to metrics definition, tool development/deployment, and reporting as well as the status of the organizing materials necessary to formally establish the organization. The Board of Directors will also review the consortium's business plan for operation and expansion during the first formal year of operation.

The core CAIDA statistics/metrics personnel will meet on a monthly basis via Internet multicast conferences, to which key technical staff from consortium members are also welcome. Members will establish schedules and detailed plans for how (and what) data to collect/analyze.

Other working groups might include the following: a) a Quality of Service (QoS) working group, focusing on audits/settlement issues; b) a Strategic Planning working group which would provide internal reports to members on emerging technologies and specific traffic patterns/issues and general reports to consumers on the state of the Internet; c) over time, an Education/Outreach working group.


5.0 Relationships with other Organizations

5.1 Existing Internet Administrative Bodies (IEPG, IETF, NANOG)

The objective of this forum is quite consistent with the spirit of the NANOG (North American Network Operators Group) and IEPG (Internet Engineering and Planning Group) groups. However, these technical organizations are informal and operate largely on volunteered resources, thereby lacking the structure necessary to implement initiatives along the lines of this proposed consortium. Furthermore, scheduled meetings of community interest groups that might develop statistics standards have hardly enough time for more critical items on the agenda, e.g., switch testing and instability of routing, to nurture the development of a consortium along the lines proposed herein.

In addition to directly engaging participants from these important organizations, consortium members will be encouraged to make regular presentations before these bodies. We also anticipate close coordination between the consortium and the IETF's Operational Area Directorate (co-chaired by Scott Bradner and Michael O'Dell) and its IPPM working group (co-chaired by Guy Almes and Matt Mathis).

5.2 Internet Service Providers (ISPs)

As mentioned earlier, several ISPs and Internet equipment vendors have already expressed commercial interest in seeing CAIDA established, acknowledging willingness to contribute shortly after its inception via gifts or labor support to CAIDA for isolated research and development tasks. MCI, BBN, AT&T, Nokia, IBM, ANS, and Cisco are among those who have indicated support.

Since CAIDA's ultimate goal is exist soley via functional support from ISPs and significant upstream providers to ISPs (i.e., Internet equipment vendors), fostering and extending these relationships will occur as quickly as possibe.

5.3 Major User Groups (Educom, FARNET, auto industry)

In addition to the federal sector, major user groups are now viewing the Internet as mission critical and are forming task groups to begin the process of defining their QOS requirements. These groups include: the Automotive Industry Action Group (AIAG) and higher education (through FARNET and Educom's Networking and Telecommunications Task Force(NTTF)). Applications developers are also placing increasing importance on their ability to monitor Internet performance as it affects their own products. Coordination with these and other representatives from the user community will be important in assisting ISPs to be more responsive to their customers' data and other related needs.

5.4 The Routing Arbiter Project

Merit/UMichigan will act in partnership with the management team of CAIDA to continue their successful advances in routing tool development and data collection activities that have been very much in the spirit of the CAIDA mission. Specifically, Craig Labovitz, currently with Merit under the auspcies of the RA project, will contribute tools for analyzing route flaps, routing instability, and long term trends in routing. Other RA-acquired data include performance statistics, delay and throughput measurements, all of which provide further insight into provider stability and network topology.

5.5 Others (Canarie, CIX, CommerceNet, ISOC, Terena, XIWT, RIPE, APNIC, etc.)

Several groups within the community are intending to move forward with measurement initiatives: CommerceNet, CIX, and internationally, Canarie and Terena. These organizations, as well as the Internet Society, the Cross Industry Working Team (XIWT) and others, have expressed interest in encouraging the definition of critical measurement metrics and the deployment of common measurement tools across ISPs. In particular, in the introduction to Internet Performance and Service Quality: User Requirements, XIWT notes that:

the ISP industry should work cooperatively, both with other ISPs in an industry-driven forum and with independent user-driven initiatives, to develop, measure, and share metrics and related information in order to continue the growth and maturation of the Internet and to improve its performance.
Since XIWT, similar to CommerceNet, is a consolidation of several large customers of ISPs, penetration to these groups will provide leverage.

Telecommunications regulators are also interested in this emerging business sector. Regulators in countries such as Singapore, for example, have defined preliminary QoS business metrics for ISPs and are beginning to measure performance.

The proposers have already been approached by Canarie, Terena, CIX, and CommerceNet regarding measurement efforts and what methodology and tools to best deploy. CAIDA tool development will progress taking into account the concerns of these constituencies.


6.0 Consortium Management

6.1 Building on NSF-sponsored community -- cooperation

In the initial stages of the consortium, there will be strong reliance on the spirit of cooperation that has characterized the (early) Internet and NLANR/RA collaboration. Once the consortium attains its status as a nonprofit, with associated bylaws and agreements, we will codify its research and data collection/analysis processes. The spirit of distributed cooperation, however, will continue as a principal characteristic of this ambitious organization.

6.2 General Grant/Subcontract Management

This grant will be managed under the guidelines set forth by NSF and by the University of California at San Diego. Dr. K.C. Claffy will serve as the Principal Investigator. UCSD will hire a Business Manager for the effort. Eric Hoffman will serve as technical lead for tool development. Via a related CAIDA proposal, Craig Labovitz (U.Michigan) will lead the routing analysis component in conjunction with the Routing Arbiter project on which he has worked for the last year. Rusty Eddy (ISI) will also assist with routing statistics analysis. Engineers for tool development and data analysis will include: Eric Hoffman (NLANR/UCSD), Duane Wessels (NLANR/UCSD), John May (LLNL), and several engineers at ISPs that are already working on tools useful for CAIDA tasks.

6.3 Business and Data Management

Business Management: In collaboration with the PI, the Business Manager will be responsible for:

  • member recruitment, including delivering presentations to potential members and other interested groups;
  • consortium organization, including being responsible for all administrative/management activities related to member relations, as well as working with members and legal advisors to draft requisite documents (e.g., charter and bylaws) and agreements (e.g., privacy agreements between participants);
  • consortium business structure, including developing/overseeing the administrative, accounting and other functions related to the organization.
Once the consortium is fully operational, an executive director will be appointed by the Board of Directors. The executive director will assume the functions of the business manager and will oversee day-to-day operations of the consortium. Funding for the executive director will be provided by the consortium membership.

Data Management: Overall management of the data acquisition, processing, and analysis will reside with the CAIDA PI and task leaders. Tasks include (task lead in parentheses):

  1. data acquisition, processing, reporting (K.C. Claffy and Eric Hoffman)
  2. analysis of IP data, including network performance, routing, and workflow characterization (K.C. Claffy)
  3. analysis of routing statistics (Craig Labovitz)

    6.4 Budget Justification

    The central mission under the proposed work is to coordinate data collection efforts of various ISPs as well as mutually agreed solutions. The rationale for government support (versus an industry-only effort) is to promote balance among the needs of the various communities (private, research, government, and users), and facilitate the near term development and deployment of critical measurement technology and techniques. We will maintain maximum flexibilty and most efficient use of funds through the use of independent contract agreements for technical support until CAIDA can sustain itself through industrial support.

    The budget includes funds for travel to conferences on Internet engineering to deliver results, standards bodies in order to participate in the formation of relevant standards, and meetings with individual organization and smaller working groups for purposes of coordination.

    We require a small number of medium range workstations and storage devices in order to perform bulk data analysis and support interactive data visualization. We will use smaller, minimal workstations (PCs or equivalent) as data collection boxes at major interconnection points and to serve as intermediate collection and digestion points.

    Because such a large portion of the proposed work involves visualization and dissemination of data analysis, we must support minimal distribution formats: in particular, we need medium grade color printing facilities and the ability to produce computer animations on videotape for widespread distribution.

    Costs to defray base level internet connectivity to appropriate provider are required as CAIDA is a distributed organization and thus cannot depending on shared connectivity costs for the prinicipal technical staff. The lowest cost option will be used to connect remote developers when other access to the internet is unavailable.

    7.0 Results from Previous and Current Awards

    The principal investigator has been working on NSF-funded projects for six years, including a proposal for network analysis (NCR-9119473) that supported her doctoral research. During graduate school and since graduation she has been working with the NSF Division of Networking, Communications and Research Infrastructure (DNCRI) under several funded cooperative agreements and grants. As a member of Hans-Werner Braun's technical staff during his tenure at SDSC, Claffy was closely involved with his focus on NREN Engineering issues in an advisory capacity for DNCRI. The cooperative nature of the association allowed us to focus on NSF mission-specific projects and proposals. In particular, they estabished the NLANR project itself (http://www.nlanr.net/), which has since considerable attention from the community with respect to networking statistics measurements and analysis issues.

    NLANR has spawned several other infrastructure-related projects: K. Claffy, Hans-Werner Braun (SDSC) and Mark W. Garrett (Bellcore) hosted an NSF-sponsored NLANR workshop on Internet statistics, measurement, and analysis (NSF Proposal/Award number NCR-9530668) (https://www.caida.org/workshops/isma/),

    One of the most community visible ongoing NLANR projects is the NLANR caching system (A Distributed Testbed for National Information Provisioning, NSF Proposal/Award number NCR-9521745). (http://www.ircache.net/), Co-principal technical investigator Duane Wessels is also involved in the statistics efforts that Caida will embrace.

    The principal investigator is also involved in a project sponsored by ACM SIGCOMM for the development of a prototype for an Internet engineering curriculum repository (https://www.caida.org/projects/iec/), for instructional materials on Internet curricula topics.


    Supporing material and recent events: (Expired Link); http://nlanr.net/Caida/supporting.html

    8.0 References

    1. Baker, F.; Guerin, R.; and Kandlur, D.; Specification of Committed Rate Quality of Service (Expired Link); Internet draft RFC draft-ietf-intserv-commit-rate-svc-00.txt; June 1996.

    2. Mitigating the coming Internet crunch: multiple service levels via Precedence , Roger Bohn, Hans-Werner Braun, K. Claffy, and Stephen Wolff, Journal of High Speed Networks

    3. Hans-Werner Braun and K. Claffy. Insight into current Internet traffic workloads. http://www.nlanr.net/NA/

    4. Hans-Werner Braun, Robert Aiken (NSF), Peter Ford (LANL) and K. Claffy (SDSC, Editor), NSF Implementation Plan for Interagency Interim NREN, Journal of High Speed Networks, Vol 2, Num 1, 1993.

    5. Hans-Werner Braun, K. Claffy and George C. Polyzos, Application of Sampling Methodologies to Network Traffic Characterization, with K. Claffy and George C. Polyzos ACM SIGCOMM '93, September 1993.

    6. Hans-Werner Braun and Yakov Rekhter (IBM Research), Advancing the NSFNET Routing Architecture, SRI Network Information Center, RFC1222, May 1991.

    7. Hans-Werner Braun and K. Claffy. Web traffic characterization: an assessment of the impact of caching documents from NCSA's web server. In Second international world wide web conference, October 1994.

    8. Brownlee, Nevil, U. of Auckland, Australia; Realtime Traffic Flow Measurement web site with references to RTFM presentations and RFC.

    9. Carpenter, Brian; Metrics for Internet Settlements (Expired Link); Internet draft RFC draft-carpenter-metrics-00.txt; June 1996.

    10. Claffy, K. Monk, T. (NLANR) Cooperation in Internet Statistics Collection (Expired Link); JFK School workshop on Coordination and Administration of the Internet; August 1996.

    11. Claffy, K. and Braun, H.W. (NLANR); and Garrett, Mark; Report of the NSF-sponsored workshop on Internet Statistics Measurement and Analysis; February 1996.

    12. Claffy, K., NLANR; Siegel, Davel, Tucson NAP; and Woodcock, Bill, PCH; A Common Format for the Recording and Interchange of Peering Point Utilization Statistics (Expired Link); white paper presented at NANOG; May 30, 1996.

    13. K. Claffy and Hans-Werner Braun, Post-NSFNET statistics collection, Proceedings of Inet '95, http://www.nlanr.net/Papers/pnsc.html, June 1995.

    14. K. Claffy and Hans-Werner Braun Network analysis issues for a public Internet, SDSC Applied Network Research group, GA-A21350, for the "Public Access to the Internet" workshop at the John F. Kennedy School of Government, Harvard University, May 1993.

    15. K. Claffy, Internet traffic characterization, doctoral thesis, University of California, San Diego, June 1994. Internet traffic characterization: a methodology to support more accurate workload characterization in the face of increasing diversity in Internet traffic types and qualities

    16. K. Claffy, Hans-Werner Braun and George C. Polyzos, Measurement Considerations for Assessing Unidirectional Latencies, Internetworking: Research and Experience, 4:3, September 1993.

    17. K. Claffy, Hans-Werner Braun and George C. Polyzos, Traffic Characteristics of the T1 NSFNET Backbone, IEEE INFOCOM'93, March 1993.

    18. K. Claffy, G. C. Polyzos, and H.-W. Braun. A parametrizable methodology for Internet traffic flow profiling, Mar 1995, IEEE JSAC Special Issue on the Global Internet.

    19. Anawat Chankhunthod, Peter B. Danzig, Chuck Neerdaels, Michael F. Schwartz and Kurt J. Worrell. A Hierarchical Internet Object Cache. Usenix, Jan 96.

    20. Cottrell, Les and Logg, Connie, SLAC; Network Monitoring for the LAN and WAN, paper presented at ORNL, June 24, 1996.

    21. Peter B. Danzig, Richard S. Hall, and Michael F. Schwartz. A case for caching file objects inside internetworks. In Proceedings of ACM SIGCOMM '93, pages 239--248, September 1993.

    22. Endicott, Don, NCCOSC; ATM Performance Workshop: Metrics, Determinants, and Statistics (Expired Link), presented at the ATM Performance Workshop; June 19, 1996.

    23. ESCC; State of the Internet Report Given at ESnet Site Coordinating Committee (ESCC), May 1996.

    24. FNCAC; FNCAC Recommendations re: Internet Statistics and Metrics (Expired Link), May 1996.

    25. High Energy Physics Community; Letter to the FNC dated March 22, 1996 on Research Use of the Internet.

    26. FNC; FNC Position Statement for the ISMA Workshop, February 1996.

    27. Monk, Tracie, DynCorp and K. Claffy, NLANR; Survey of Internet Statistics/Metrics Activities, presented at meetings: Commercial Internet Exchange (CIX) (24 Jun 96), Coordinating Committee for Intercontinental Research Networks (CCIRN) (29 Jun 96), Montreal, Canada.

    28. Moskowitz, Robert; AIAG Position Statement for ISMA Workshop, February 1996.

    29. Paxson, Vern; Lawrence Berkeley National Laboratory; Towards a Framework for Defining Internet Performance Metrics, presented at INET '96, June 1996.

    30. John Romkey, editor, Report from the October 1994 Internet Architecture Board (IAB) information infrastructure workshop

    31. XIWT (Cross-Industry Working Team) Internet Performance and Service Quality: User Requirements

Related Objects

See https://catalog.caida.org/details/paper/1996_caida/ to explore related objects to this document in the CAIDA Resource Catalog.