1. Overview

Introduction:

In this paper, we present a survey of the current activities in Internet performance and workload measurement. The environment is dynamic enough that we can only provide an indicator of the growing activity and interest surrounding this important topic; we apologize for anyone omitted. Hopefully, through the hyperlinks and other references, readers will be able to followup directly with the various researchers and organizations who have (or plan) initiatives relating to the collection, analysis and/or presentation of Internet statistics.

We also outline in this paper a recommendation for the establishment of a `provider consortium' to facilitate cooperation. The National Laboratory for Applied Networking Research (NLANR) is working to develop a framework for such a collaboration, and believes that it could serve as an appropriate forum for...

• facilitating the identification, development and deployment of measurement tools across the Internet;
• providing commercial providers with a neutral, confidential vehicle for data sharing and analysis;
• providing networking researchers and the general Internet community with additional realtime data on Internet traffic flow patterns; and
• enhancing communications among commercial Internet service providers, exchange/peering point providers and the broader Internet community.

Transition from the NSFnet Backbone Service:

Throughout the lifetime of the NSFnet Backbone Service, Federal agencies and various user communities were engaged in the development, operation and ultimate transition of the Internet to the commercial sector. The involvement of many of these groups was channeled through the Federal Networking Council (FNC) and its Advisory Committee (FNCAC), which consists of representatives from the higher education and research communities, service providers and industry. Following the April meeting of the FNCAC, its members distributed a set of recommendations calling upon government to...

• Initiate efforts to expand the collaborative development of performance measurements and trouble ticket tracking on the Federally-sponsored segments of the Internet, with specific attention to factors relating to the privacy of Internet users and providers and security of Internet facilities and usage.
• Increase funding / prioritization of research on measurements and measurement techniques that can be employed by ISPs and users (or their representatives) to quantify Internet quality of service (packet loss, packet delay, route availability, etc.).
• Identify critical networking metrics and tools which could be run over Federal R&E networks, including defining the characteristics of an "ideal" measurement tool which could gather data on both end-to-end performance and workflow characterization.

These recommendations resulted from concerns expressed by the members' communities (such as the High Energy Physics community) about the current state of the Internet as well as briefings from researchers such as Hans-Werner Braun (Teledesic - previously NLANR), K Claffy (NLANR), and Mark Garrett (Bellcore). The most recent presentation before this body concerned results from statistics gathering efforts over the FIX-West facility and the conclusions of the Internet Statistic and Metrics Analysis workshop. The results of this workshop and other statistics / metrics activities are discussed below.

2. The Demand for Metrics

ISMA Workshop:

In February 1996, NLANR and Bellcore hosted an NSF-supported workshop entitled Internet Statistics and Metrics Analysis (ISMA). One outcome of this workshop was the articulated need for smoother coordination and information exchange among Internet service providers -- both providers of Internet access and traffic exchange services.

Although most participants at the workshop felt that the Internet development and provider community should seek out measurement infrastructure and sources of statistics in the commercially decentralized Internet, there was definite dissonance as to which measurements would help, and who should have access to them. While a public infrastructure for end-to-end performance measurements could help researchers and end users study the infrastructure, participants concurred that Internet service providers (ISPs) themselves would be the greatest beneficiaries of an enhanced statistics capability. Opinions varied as to the sensitivity of some data and how much could be released publically. But there appeared to be consensus supporting the need for desensitized versions of such statistics to be made available in a neutral forum. Such a neutral forum could also facilitate more comprehensive collaboration, consensus-building, and the development of tools to measure new metrics that are important to stability, service, efficient resource allocation, and more economically viable network usage pricing policies.

Financial Settlements

The Internet is still relatively devoid of pricing models or other mechanisms to allocate and prioritize scarce resources -- particularly bandwidth. Enhanced network functionalities such as multiple qualities of service and bandwidth reservation systems such as Resource Reservation Protocol (RSVP) will require some mechanisms for accounting for network usage at a finer granularity than is generally deployed, or even possible, within many operational Internet components today. They will also require basic statistics to facilitate accounting and the development of economic models.

In his recent Internet Draft, on Metrics for Internet Settlements, Brian Carpenter (CERN), asserts that financial settlements are a critical mechanism for exerting pressure on providers to strengthen their infrastructures. He writes:

"...the current crisis in Internet performance, although superficially caused by very high growth rates, is to a significant extent due to the lack of efficient financial pressure on service providers to strengthen their infrastructure. One way to fix this is to institute financial settlements between the providers."

He suggests that metrics used in Internet settlements

...should not rely on expensive instrumentation such as detailed packet of flow analysis -- simple bulk measurements on a wire, or off-line measurements, are preferable.

...should preferrably be estimated, if necessary, by statistical sampling.

...should be symmetric in nature so that the resulting settlements can be associative and communicative, thereby allowing settlements to be aggregated by upstream service providers, and redistributed by transit carriers, in an equitable manner.

Carpenter further suggests that each metric used in the settlement agreement should define or refer to a measurement method and should specify a settlement rate and currency.

Factors which continue to inhibit implementing settlements include the lack of a common understanding of the business mechanics of inter-ISP relations. Some suggest that the ITU / telco settlements model may have relevance to ISP settlements. Others suggest that the connectionless nature of the IP protocol demands that entirely new pricing models be developed. For a good introduction to discussions on Internet economics, see Hal Varian's web site on The Information Economy.

User Communities:

The higher education and research communities are among the more vocal Internet users. Being among the first to find the Internet mission critical, they are quite chagrined with the current state of congestion -- and with the fact that the Internet (given its design) offers only best- effort guarantees. They claim that service quality over recent months has been significantly degraded due to the growth in web activity, the proliferation of users (see Mark Kosters' May 1996 presentation) and the lack of cooperation among commercial providers / users which exemplify the current Internet environment.

In a series of meetings over the last year, representatives from EDUCOM, FARNET, and related institutions have discussed the communications requirements of their communities and shared their concerns about the ability of the Internet to meet their future needs. A paper by Doug Gale and the Monterey Futures Group, concludes "that by the year 2000,higher education will require an advanced internetworking fabric with the capacity to:

• support desktop and room-based video teleconferencing;
  
• support high-volume video streaming from distant video servers;
  
• incorporate and integrate voice traffic;
  
• insert large-capacity inter-institutional projects into the fabric at will in response to the needs of research projects and network experiments;
  
• interconnect enterprise networks, that are in various stages of migration to higher performance technologies, and;
  
• control costs and pricing/allocation models from within the enterprise."

Similar demands are beginning to be made by other user groups who now view the Internet as the appropriate medium for their future communication needs. The automotive industry, for example, adopted TCP/IP in 1995 as a standard for data communications among its thousands of trading partners. Through the Automotive Industry Action Group and its Telecommunications Working Group, the major auto manufacturers are working to set forth necessary requirements for the quality of service necessary to satisfy their industry's needs. Specific areas being examined include specifications related to...

• certifying a small number of highly competent Internet service providers to interconnect automotive trading partner's private networks;
• monitoring providers' ongoing compliance with performance standards that support business requirements;
• enforcing strict security mechanisms to authenticate users and protect data, thereby creating a virtual private network for the auto industry

The AIAG has identified several metrics which it views as critical to this initiative and to future monitoring efforts. These include Performance Metrics such as latency, packet/cell loss, link utilization, throughput, and BGP4 configuration and peering arrangements; as well as Reliability Metrics such as physical route diversity; routing protocol convergence times; disaster recovery plans; backbone, exchange point and access circuit availability; and speed of failed customer premise equipment replacement.

As more user groups (e.g., the financial sector, energy industry, and others) move toward the Internet as their preferred communications vehicle, we are likely to witness increasing pressure on providers and others to collect, collate, analyze, and share data related to Internet (and provider) performance.

Instrumenting Networks:

Many internet service providers currently collect basic statistics on their own network's performance and traffic flows. Typically this includes measurement of throughput, delay, and availability. In the era of the post-NSFnet Backbone Service, the only baseline against which networks evaluate performance is their past performance metrics. There are no data available against which national level comparisons or comparisons with other networks' performance can be made. Increasingly, what is needed by both users and providers is information on end-to-end performance -- information which is beyond the realm of what is controllable by individual networks.

Another example of statistics maintained in isolation within an individual ISP is trouble ticket tracking of problems that originate and are resolved within the context of that ISP itself. Throughout most of the lifetime of the NSFnet backbone, resolving route instabilities and other trouble tickets were the the responsibility of Merit (under its cooperative agreement with NSF). In the current environment there is no such entity to claim to share responsibility for national level management of the Internet. As a result, there are no scalable mechanisms available for resolving or tracking problems originating or extending beyond the control of an individual network.

Route instabilities is another area which can have a direct, sometimes profound, affect upon the performance of individual networks. Some networks are seeking to improve the stability of their routing by peering directly with the routing arbiter (RA) at network access points (e.g., SprintNAP and FIX-West/MAE-West). In the context of the routing arbiter project, Merit/ISI have also developed statistics, including those on route flapping and inappropriate announcement, that represent a macroscopic characterization of routing stability and identify trouble areas for the networks with which they peer. However, these efforts are still in their nascent stages and do not yet have sufficient buy-in or support from commercial players to make them a fundamental component of the Internet architecture.

The vacuum created in national-level statistics/metrics collection which followed the transition to the commercial architecture has also complicated planning by national service providers and others. While detailed traffic and performance measurements are essential to identifying the causes of network problems and formulating corrective actions, it is trend analysis and accurate network/systems monitoring which permit network managers to identify "hot spots" (overloaded paths), predict problems before they occur and identify ways to avoid them by efficient deployment of resources and optimization of network configuration. As the nation and world become increasingly dependent on the National and Global Information Infrastructures (NII/GII), it is critical that mechanisms be established to re-enable infrastructure-wide planning and analysis.

Many of the metrics above are inherently problematic for the Internet infrastructure, and still require critical research. At INet '96, Vern Paxson' paper on Towards a Framework for Defining Internet Performance Metrics (compressed postscript here) will provide a cogent discussion of the need for new tools for passive and active measurement of network traffic. As these tools are developed, particular attention needs to be devoted to privacy considerations (particularly with passive measurement tools) and to designing tools which are minimally invasive (particularly on active measurements where the tools themselves can potentially disrupt or distort traffic patterns). Additional attention should also be devoted to the problem of routers dropping packets and how to measure this phenomenon.

A list of some emerging tools / initiatives which should facilitate IP analysis is provided below:

• Network Probe Daemon (NPD) (Vern Paxson, LBL). For end-to-end measurements across a representative cross-section of the Internet. Examines route stability, packet loss, queing levels, buffer sizes, and available bandwidth. Must be installed at remote nodes. Utilizes traceroute, TCP probe, and tcpdump.
• Hop Congestion Tool (E. Hoffman, CAIDA) is a tool under development that incorporates traceroute and ping functionality to build a database of delay data along paths to a specified set of destinations, and visualizes the resulting data.
• Treno (Matt Mathis, PSC) Uses ICMP and TCP packets to measure throughput and congestion. INet '96 paper at http://www.psc.edu/~mathis/papers/inet96.treno.html

  NetSCARF (Bill Norton, Merit) New project to develop a public domain network statistics collection and reporting facility that will evolve to meet the needs of ISPs. Funded by the Resource Allocation Committee, the first software release is expected in July 1996. Software will collect statistics necessary for individual networks to display graphs on total number of bits, system up time, and interfaces links. Current plans are to collect at 15-minute intervals, and display results by individual network. No near-term aggregation of data from multiple networks is expected. For information, contact Bill Norton (wbn@merit.edu) at Merit.

• Mix (Dave Siegel, RTD Systems/Tucson NAP; Bill Woodcock, PCH; and K Claffy and Duane Wessels, NLANR) have developed a common format for recording and interchange of peering point utilization statistics. The current efforts are focused on inbound traffic collected at each router. However, there are also opportunities (and a potential demand) for statistics related to: packets and bytes in/out per participating organization; collisions on any shared-access media segment; discards on any packet-switched segment; cell-loss on any cell-switched segment; and mbone, ftp, telnet, http, nntp, dns, BGP, icmp and snmp as percentages of total throughput. For more information, contact either Siegel (dave@rtd.net) or Bill Woodcock (woody@zocalo.net).
• tcpdpriv (Greg Minshall, Ipsilon) Takes packets captured by tcpdump and removes or scrambles data within the packet to protect privacy (e.g., source and destination hosts and ports). For more information, contact Minshall at (minshall@ipsilon.com).
• Future data collection/analysis activities will likely stretch beyond the realms of a plain packet switching infrastructure, toward optimizing overall service quality via mechanisms such as information caching and multicast. Visualization is important for making sense of all the data sets described above, and is especially critical for developing and maintaining the efficiency of logically overlaying architectures, such as caching, multicast, mobile, IPsec, and IPv6 tunnel infrastructures. As examples, NLANR has created visualization prototypes for the mbone tunnel logical topology and the NLANR web caching hierarchy topology (Expired Link). For the latter they have also built a tool for automatically generating a nightly update of the caching hierarchy topology map. Such visualization could be of great assistance for improving the global efficiency of traffic flows.

• At the May 1996 NANOG meeting and during the BOF on May 30th, several related topics were discussed, including those by Alan Hannan (Global Internet) associated with Inter-Provider Outage Notification; Craig Labovitz (Merit) on the ASExplorer and Other New RA Tools; and by Steve Crocker (CyberCash), who called for a consortium of users who would define requisite metrics and QOS and ensure collection of appropriate performance statistics.

ATM:

ATM performance is receiving increasing attention. Kansas University is planning a workshop on this topic on June 19-20, 1996. The event is hosted by Kansas University (contact Victor Foster at frost@tisl.ukans.edu), with sponsorship by DARPA and NCCOSC. For an excellent overview of the subject, see Don Endicott's slides (Expired Link) from this meeting.

Kansas University has also worked with the Department of Defense and Sprint to develop the Netspec tool for characterizing a range of interacting and independent loads on the ACTS ATM Internetwork (AAI) project's network. The Netspec web site also discusses the performance of this tool compared to other commonly used network performance testing tools, such as NetPerf, nettest, and ttcp.

A new RFC by Steven Berson (ISI) has just been released which provides guidelines for using ATM virtual circuits (Vcs) with QoS as part of an Integrated Services Internet. Other groups working on ATM tools include the ATM Forum (www.atmforum.com) and Bellcore.

5. The Need for Cooperation

Provider Consortium:

A limited exchange of statistics should have direct payback for ISPs in achieving their goals such as configuring and managing their networks, and even greater payback to the Internet community by enabling analyses which can strengthen the overall information infrastructure.

Market Pressures upon ISPs to participate in such a consortium concept include:

• Providers' customers are increasingly relying on the Internet for "mission critical" applications and demanding that the Internet's infrastructure be strengthened
• QoS / RSVP / Internet Settlements will require availability of authenticated and possibly confidential provider statistics across the Internet
• No single company can do it all -- the nature of the Internet as a web of interconnected networks suggests that organizations should collaborate and cooperate if the underlying structure is to be improved. An excellent model for such collaboration is provided by the Multimedia Services Affiliate Forum (MSAF) which was formed in February 1996. MSAF provides a global forum for advanced inter-networked business services and consists of the world's largest telecommunications providers and multimedia service firms.

Business Constraints hindering such cooperation include:

• the competitive Internet business environment
• data privacy considerations
• concerns about the appearance of industry collusion by major providers
• the continuing perceived gap between the needs of providers' marketers/managers and network engineers
• the lack of adequate pricing models and other mechanisms which could insert a sense of economic rationality into the process

Technology Constraints hindering the collection and analysis of Internet metrics include the facts that:

• Measurement tools are in a nascent stage of development
• New and emerging technologies, e.g. gigaswitches and ATM, significantly complicate data acquisition/analysis

Despite these challenges, the requirements for data collection, analysis, and distribution exist and will have to be addressed at some point in the near future. Collaboration toward this end is critical.

Next Steps:

Developing an effective provider consortium will require (minimally):

• Participation by 3 or more of the major service providers, e.g., ANS, AT&T, BBN Planet, MCI, Netcom, PSI, Sprint, or UUNet.
• Establishment of a neutral third party with sufficient technical skills to provide the core data collection and analysis capabilities required by the consortium.
• Development of appropriate privacy agreements to protect the interests of members.
• Agreement on which basic metrics to collect, collate/analyze, and present.
• Agreement on which new tools need to be developed -- particularly those related to emerging infrastructures using ATM, gigaswitches, and other new technologies.