Contents
Background
Introductions (Mark Garrett, Hans-Werner Braun)
Meaningful Internet measurement (Vern Paxson, Walter Willinger)
Measurement infrastructure (Vern Paxson, Guy Almes, Nevil Brownlee, Robert Moskowitz)
Flow characterization and control (Craig Partridge, Peter Newman, Joel Apisdorf, K Claffy)
Scaling (Dennis Ferguson, Mark Garrett)
Quality of service/pricing (Roger Bohn)
Where to go from here (Walter Wiebe)
Privacy and security (Carter Bullard)
How and what statistics will help (Mark Garrett)
Federal perspective (Mark Luker)
Epilogue (post-mortem reflections)

ISMA '96 workshop home page
ISMA '96 workshop agenda
relevant materials

Attendance at the workshop was limited, and attendees were expected to present or formulate position papers to participate. The workshop was organized by K. Claffy, Mark Garrett, and Hans-Werner Braun. This report is derived from minutes taken by various contributors at the workshop.

This material on this report is partially based on work sponsored by the National Science Foundation under NSF grant No. NCR-9530668. Any opinions, findings and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or other participating organizations.

In the first half hour attendees introduced themselves and described their role and interest in participating. The group included representatives from several interest groups: ISPs, NAPs, vendors of both IP and ATM equipment, researchers, and government. Hans-Werner Braun then presented briefly on the scope and purpose of workshop. The agenda focused on both technical and policy aspects of traffic measurement: what Internet problems need solutions that require measurements, what kind and where they need to occur, and what are the obstacles to collecting and making the data available to the research community on a continuing basis. He expressed an interest in seeing, as a result of this workshop, a specification of a minimal set of metrics conducive to use in a competitive environment where traffic traverses many service providers en route to its destination.

Braun focused on several pressing needs:

• accountability for resource consumption
• understanding of network workload profiles
• service architecture from the end user perspective
• metrics of brokenness

The disparity in workload profiles in the current cross-section of Internet applications necessitates revised metrics for describing network behavior. Simple mean or peak utilization figures are ineffective in addressing a service provider's engineering needs, without also knowing the transaction profile constituting and perhaps dominating those figures. Keeping track of workload profiles requires measuring flow data at relevant network locations. (NLANR currently supports operational collection of such data at several points, including data from the FIX-West multiagency network interconnection facility.)

More accurate resource consumption, and concomitant pricing models, will allow progress with another severe need in the current infrastructure: a service architecture from the perspective of the end user. As we will discuss later, communities of users are now frustrated enough with the current quality of their Internet service that they are seeking or trying to develop independent third-party mechanisms for measuring and verifying service. ISPs, while perhaps not fond of an effort to develop metrics to rate their service, will likely find it in their best interests to participate in the process rather than ignore it.

Braun noted several factors leading to the current Internet environment, which though at one time was amenable to modeling as a circuit problem, may now be more reasonably attended to as a biology problem. The current economic structure of the market, devoid of rational cost models, contributes to the overwhelming demand that ISPs race to meet, leading to the growth we see threatening global stability. Time for network analysis is scant, typically devoted to the first and third of three separate strains:

• 24x7 network monitoring/management
• medium term one-shot engineering studies, i.e., workload characterization and performance evaluation
• crisis management, i.e., in the face of network meltdowns, perhaps due to security violations, misconfigurations, machine crashes, protocol interoperability

Mark Luker, current NSFNET program manager, offered a description of what NSF would like to see come from the workshop. He hears increasing feedback from within the R&E community that their work requires more reasonable service that the Internet can now provide. Simply adding capacity is not a solution, since R&E connectivity now involves multiple service providers, none of whom can add capacity as fast as their customers can consume it. In addition, they need different kinds of service, not available before, for new applications. Luker also announced that NSF would soon release a New Connections program solicitation for high performance R&E Internet connections, where meritorious science and engineering projects would receive bandwidth resources above and beyond what is commercially available. (The NSF released the solicitation a few weeks after the workshop, mid-March 1995.) Proposers would negotiate with their campus providers, ISPs, and the NSF vBNS, on how to secure high performance for such projects.

Vern Paxson of LBL presented his view of why meaningful Internet measurement is painfully hard, starting with a list of several as yet unanswered questions that require Internet measurement:

• how many people use the Internet?
• what does a typical connection look like?
• what is the dominant application?
• how many packets are dropped?

Walter Willinger of Bellcore presented a complementary view on why there is hope for meaningful Internet traffic measurements. He gave the good news: Although Internet traffic does not exhibit Poisson arrivals, the cornerstones of telephony modeling, number of researchers have measured a consistent thread of self-similarity throughout all of his studies from a variety of networks, including samples of SS7, ISDN, Ethernet and FDDI LANs, backbone access points, and ATM traffic. Self-similarity can derive from the aggregation of many individual, albeit highly variable, on-off components. Self-similarity implies that a given correlational structure is retained over a wide range of time scales. Although self-similarity is a parsimonious concept, it comes in many different colors, and we only now are beginning to understand what causes it.

The bad news about self-similarity is that it is a significantly different paradigm that requires new tools for dealing with traffic measurement and management. Load service curves (e.g., delay vs. utilization) of classical queueing theory are inadequate; indeed for self-similar traffic even metrics of means and variances indicate little unless accompanied by details of the correlational structure of the traffic. (The technofolks in the room got off on a bit of a self-similarity tangent, basically trying to address anecdotal data from backbone providers.), but there was strong agreement that answering their questions and refining the models toward greater utility for network engineering would require traffic measurements from a wide variety of locations.

Mark Garrett summarized several points of the discussion:

1. engineering without measurement is dangerous,
2. new models (theory) and data are needed,
3. the federal government should encourage and consider funding projects that leverage implications of recent theoretical and empirical work
4. we should seek out measurement infrastructure and sources of statistics in the commercially decentralized Internet

In transition to the next topic the group discussed existing sources of Internet traffic measurements. Last year Paxson et al led the establishment of the Internet Traffic Archive (ITA), and the associated moderated discussion list. The ITA holds traces as well as software for manipulating them. Other sources of traffic information are few and far between, mostly due to ISP privacy concerns. The few ISPs that do maintain statistics are either unable to release them due to customer privacy concerns, or unwilling to share them since they reveal competitive information.

There are even ISPs who consider customer privacy so critical, and any statistics collection at all a violation of it, that they refuse to track customer data in any way. Bill Schrader of PSI eloquently articulated this position in his response to the invitation to the workshop. Although there is some sketchy legal framework for traffic monitoring for engineering purposes, this legislation does not extend to making any such information public. However, the integrity of the Internet may depend on balancing the needs for data analysis with this recognized sensitivity to privacy issues. To support this balance, researchers themselves have developed tools to encode address information while retaining the structure needed for realistic investigations.

Regarding the latter problem of disclosing data that could help competitors, Sean Doran of Sprint noted that he does not feel any competitive threat in the public Internet market, since there is more than enough demand to share for the foreseeable future. Most ISPs are barely managing to keep up with demand. Indeed, for an ISP that also leases private lines, the uncoordinated public Internet is a great business generator, since it requires all competitors to run a decent network to ensure connectivity. Unlike the public Internet, where demand is overflowing, the market for leased lines is competitive, and thus there is no incentive to cooperate with other ISPs with whom one also competes for leased line business. Since installing leased line provides a higher return than cooperating with other ISPs, the non-cooperative strategy dominates, which hardly fosters a healthy global Internet, or even leased line customers with decent connectivity beyond their private trunks. And yet there is little attention among many ISPs to the health of the global Internet beyond the extent that it affects their core business. If a customer of one ISP cannot reach a remote site due to a problem several ISPs away, he can tell his own ISP but the chances of effecting resolution are small. Customer support is often only intra-ISP, not inter-ISP, a choice derived from limited resources and lack of supporting technology rather than disinterest. Cooperative forums such as NANOG or IEPG might be reasonable places to develop cross-ISP trouble reporting mechanisms, but most ISP engineers are too busy to actively participate in these forums beyond just attending a few meetings a year.

Paxson of LBL presented his work on black-box measurement of Internet clouds, using endpoint measurements to discern how an Internet cloud perturbs traffic going through it. He found 35 sites to participate, enabling him to procure data on a wide range of links. (Note that each additional site introduces N additional Internet paths to probe and analyze). He then used his traceroute-based tool, npd (network probe daemon), to track route stability and failure patterns over time. His measurements taken over the course of a year indicate a significant service degradation in the past year (e.g., a substantial increase in the number of outages of 30 seconds or more). He also found widespread asymmetry (the different directions of an Internet path visit at least one different city about half the time). The software has been ported to several unix platforms, allowing widespread deployment of such probes. Vern notes that npd was just a prototype he wrote for his thesis research, although he agreed that obtaining a reasonable picture of performance would be greatly facilitated with some measurement infrastructure, including deployment of performance evaluation tools, techniques for independent measuring and analyzing of of available congestion levels, queueing delays, packet loss, etc.

Nevil Brownlee presented data from metering software used in New Zealand's Kawaihiko University network. The NeTraMet software consists of meters that watch packets and build tables of information about flows, meter readers to gather flow data, a manager to coordinate meters and collectors, and a customizable rule configuration file. An IETF working group on realtime traffic flow measurements (RTFM) and mailing list (subscribe to rtfm list via majordomo here), focuses on the use of such software. Nevil described the operational environment in which he uses NetraMet: a university network with a full mesh of PVCs. There is a meter at each of seven universities in New Zealand, with about 600 rules each, split between commercial and non-commercial traffic. The software allows him to create a traffic matrix.

Since he started weekly time-series and utilization plots, he has not seen a significant degree of aggregation; there are still heavy tails. he has noticed that the Auckland data is a lot `spikier' than that from most other sites. One explanation for the difference is that Auckland's access rate and frame relay CIRs are a lot higher than its average load, so there is no artificial `trimming'. The other universities tend to run their PVCs closer to their CIRs. Over the last month or two Canterbury has increased its CIR significantly, and their time-series plots seem to have got spikier.

Guy Almes of Advanced Network and Services spoke next, on a complementary IETF working group effort: Internet Provider Performance Metrics (IPPM). Their charter is to develop and deploy objective measurement tools for use in the post-NSFnet architecture. Customers have a problem in trying to select an ISP: their only metric is the price tag, perhaps in addition to rumors about performance. More rational selection requires performance metrics that are concrete and repeatable, exhibit no bias for identical technology or artificial performance goals, and which independent users can rely on to evaluate performance.

To clarify the distinction between the RTFM and BMWG/IPPM working groups, RTFM focuses on network-centric workload characterization of longitudinal traffic measurements, (bit counting), while BMWG/IPPM focuses on measuring performance (delay, throughput, loss), treating background traffic as an exogenous variable. Both groups focus on objectives different from what the SNMP or RMON network management protocols cover. Fred Baker of Cisco offered to provide a half-page summary of RMON and RMON-II.

Guy discussed end-to-end path measurement methodologies. Two path performance metrics of interest are delay and flow capacity. Statistics of delay as measured by ping are somewhat useful (minimum, maximum, mean, variance, percentiles), and one-way vs. round trip path measurements are increasingly relevant in the face of asymmetric routes. Measuring flow capacity is problematic because determining how much capacity one can get from a given path ostensibly requires ramping up traffic flow across the path until degradation occurs. Matt Mathis has written treno, a tool to measure flow capacity in a somewhat controlled way.

Guy discussed the need to be able to estimate values in one area using combinations in another area, or using previous measurements of delay and flow capacity, to determine current or predict imminent flow capacity. Such extrapolations are still an area of research, but probe computers, or transponders at key sites, i.e., exchange points, regional provider gateways to backbones, campus gateways, are still needed to develop more reasonable models, in particular a model that allows us to ascertain performance numbers without saturating the measured networks.

ISPs are in general skeptical of performance measurements. Steve Corbato from the University of Washington compared the situation to the Mbone: ISPs have begun to introduce Mbone tunnel support since it became obvious that their customers would just set up tunnels anyway, and that without coordination by the ISP, the result would be much less efficient, with as much negative impact on the ISPs as on anyone else. Network flow capacity testing poses the same threat: users will develop and deploy tools and metrics themselves if no better methodology is available. Furthermore, without a vehicle for sharing the data there will no doubt be unnecessary redundancy, not to mention inaccuracy, in effort, so we might as well codify and consolidate useful efforts, and develop benign tools and well-engineered architectures for probe machines. We might as well do it right. Bill Norton noted that NSF would be likely to fund a packaged turnkey system that ISPs could use to offer standard graphs (e.g., a la NSFNET) to the community.

The conversation came back to determining a simple and useful Consumer Reports style (though the phrase was not universally liked) service metric, e.g., periodic ftps and pings. Sean Doran warned that the last thing that high quality ISPs need is for Consumer Reports to endorse them and cause everyone to shift traffic over to them. Sincoskie and Braun felt this concern merely justified continually repeated measurements and credible criteria.

The next speaker was Robert Moskowitz of Chrysler, who is leading the Automotive Network Exchange, an advocate organization on behalf of North American automakers in pursuit of a publicly endorsed mechanism for measuring the ability of various ISPs to meet their requirements. In particular, the industry wants to be able to dictate terms to ISPs, trading partners, and then monitor or pay an independent agency to monitor them. Performance metrics will include per time period measurements of total data transferred, and average, peak, and normalized utilizations. Reliability metrics, e.g., circuit availability, were even more important to the auto industry. Moskowitz noted that a 3-minute outage can bring down an assembly line; after 20 minutes the plant may shut down and the shift sent home. Help desk metrics include: total calls/period, mean time to repair (MTTR), and 95th percentile worst time to repair. The ANX intends to publish statistics similar to how the FAA publishes airline ontime performance.

Sean Doran, apparently reflecting the opinion of a few other ISP representatives in the room, was perplexed by Robert's unrealistically ambitious objective: the current Internet, including underlying router technology (not just service quality, though one can criticize that too) is simply not amenable to the control that Moskowitz's agenda requires. Routing instability and convergence times, especially when crossing large ISP interchanges, inhibit that kind of integrity. If he wants a guaranteed 3-minute reachability repair time, Moskowitz needs a set of private interconnects, a single-ISP solution that avoids exchange points, so he has a single point of contact/liability for connectivity problems or violation of contractual agreements. Both Sprint and MCI voiced willingness to meet these needs with leased lines. Moskowitz assured the group that they had considered this option, but given that the ANX needs all up and downstream suppliers and dealers connected too, with the same service quality, and that since one in seven working Americans have automotive related jobs, this approach would just constitute a reconstruction of the Internet. Why not fix the one we have?

Moskowitz was not arguing that the public Internet could provide what he needed today, or even this year. His goal was rather to develop mechanisms to achieve his vision as soon as possible, and the obvious place to start is developing standardized metrics from which to compare the ability of different ISPs to meet an industry's networking requirements. Several participants noted that Moskowitz was not alone; other customer communities, most notably higher education, had voiced precisely the same concerns, and would also soon begin to systematically measure ISP performance with or without participation of the ISPs themselves. It is thus quite likely in the best interest of ISPs to participate in defining such measurements and metrics in a neutral and productive forum. Without ISP and IP research community participation, bogus measurements and misleading metrics may result, leading to inaccurate evaluation of ISPs and unhappy customers.

Over lunch, kc showed a demonstration video (Planet Multicast), of a recent network visualization tool for examining a particularly problematic Internet component: the Mbone. Developed by Tamara Munzner at Stanford graphics department, the tool lets you interactively examine and move various components of the Mbone topology. A brief description of the work and resulting VRML and Geomview-compatible images are on the CAIDA web server.

After lunch, both Craig Partridge of BBN and Peter Newman of Ipsilon Networks presented statistics on the potential performance of flow caching for a September 1995 5MB, 30-minute packet trace from FIX-west. Both are working on high-speed IP router projects at their respective companies, and flow duration statistics that can assist in deciding which flows should be switched are critical to optimizing router performance. Partridge found that even with only modest hit rates, as low as 10 to 20%, caching can be effective in improving performance. Their analyses indicated that by recognizing traffic suitable for fast switching (caching) and treating it accordingly, one can greatly increase effective switching capacity, more than 5-fold in Newman's simulations on this specific trace. Newman and Partridge both emphasized that further developing and establishing the validity of such technology requires raw packet traces: headers and accurate time stamps, with sanitized addresses that retained the structure of CIDR prefix information.

Apisdorf and Claffy presented a real-time traffic collector, a prototype of which Apisdorf has recently ported to an OC-3 ATM-based monitoring platform. The vBNS and other infrastructures could not make reasonable use of the original FDDI-capable NNStat-based software because they needed to monitor the fiber coming directly from the ATM switch at each backbone node. Apisdorf has ported the collection architecture and code to a 120 MHz, 64MB Pentium PC; both will be public domain. The monitor can capture all of both TCP/UDP and IP headers if IP options are not used, and they expect it to keep pace with full OC-3 rates. The monitor is not operational yet on the vBNS; they demonstrated it on the San Diego node, with the same graphical interface for customizable graphs as they use for the FIX-west statistics.

The Digital Alpha lent by MCI for this data collection project collects 5 minutes of complete packet data (mod losses) every hour on the FIX-west FDDI, calculates flow-based statistics on each 5 minute trace, and transfers the results down to the NLANR web site which supports a real-time interactive statistics querying engine. Because the FIX-west FDDI monitor had been running for several months, Claffy interactively created a few interesting time-series graphs as well as comparing current Internet traffic with that on the SDSC vBNS node. In particular, the tool allowed one to show the disparity in bandwidth and switching resources consumed versus customers served (i.e., flows) using web and CuSeeme traffic profiles. For example, an interactively created graph of the fraction of total packets versus the fraction of total flows illustrated how CuSeeme traffic satisfied few customers with a significant consumption of resources for each one.

Claffy warned that this source of statistics will disappear when FIX-West converts to a Gigaswitch platform, planned in June 1996, after which accomplishing the same task would require the prohibitive cost and complexity of separately monitoring each attached FDDI and collating the statistic sets. Doran noted that SprintLink had 150 routers and 21 were running similar flow statistics gathering software on the Cisco 7513 platform. Partridge suggested that we assume that every counter implemented costs 3% of router performance, and Sincoskie emphasized the importance of being able to download analysis code to a router, so the user (ISP) can decide his own tradeoff between routing packets and gathering statistics. Joel Halpern still wanted a better idea of what to collect in the router, to ensure that the data reduction is useful later.

Even if we had such boxes, at which strategic points would we want to deploy them? An increasing number of direct interconnections among service providers has rendered the NAPs less significant for anywhere but the R&E environments, although they are still the only place that NSF may have any leverage. Mark Luker of NSF acknowledged that they were considering such issues, and were open to input on where such probes are needed. Although NAPs, campuses, ISP backbones, local providers are all important to measure, the NAPs may be the most difficult to get consent for the collection from the attached providers. Most of the group agreed that aggregated information was safe to distribute, and further anonymity could be secured through appropriate encryption of addresses. (Carter asked about the impact of the Telecommunications Act of 1996, but noone seemed to want to even discuss it.) Nonetheless if NSF were to make anonymous monitoring a part of the NAP agreement, many providers would just disconnect and set up their own peering points. Braun did not have the same problem with the FIX-west data, since not only was it totally aggregated and sanitized, but there were no legal proprietary issues since all connections to the FIX are specifically approved and in the service of U.S. federal agencies.

Hon So of the Pac Bell NAP urged us to make it clear to ISPs of the benefit of allowing their traffic to be monitored. Luker agreed, suggesting to use sample measurements to make a strong economic case that it would improve the price-performance of their network.

Dennis Ferguson presented reflections from a former backbone operator on the two most acute needs in the Internet right now: higher performance (`rrrreally big') routers, and a basic traffic engineering methodology.

He first described the suboptimality in trying to use many existing (i.e, mid-performance, since high performance ones do not exist yet) routers to achieve the route processing and convergence time requirements of high bandwidth environments. He took a dimmer view of route caching, confident that the backbones will always outgrow the existing route cache algorithms. His projection for IPv4 routing table space was for 350,000 routes before IPv6 would establish itself, thus a a minimal target for IPv4 router vendors.

Ferguson then described the other Achilles' heel for ISPs: in 10 years there has not yet developed a basic Internet traffic engineering methodology. If even half the attention to rocket science traffic modeling were devoted to how to estimate a reasonable ingress-egress traffic matrix from sampled measurements, network engineers, particularly of large clouds, would find their job substantially easier. And yet, not even matrix estimates are available with most router equipment right now; link loads are available, but they just do not hold enough information to estimate a reasonable matrix. He wanted to know how to derive better estimates from samples and averages, the way civil engineers have derived approximations for highway architectures.

Sincoskie responded that we lack the background to develop models with what we have; further study, e.g., of self-similarity is required. Ferguson assured us that he does not need precise models right now; he needs realistic short-term engineering guidelines. His many observations of backbone traffic are mostly inconsistent with self-similarity models anyway, a rough traffic matrix would at least improve his ability to balance load and engineer topology.

Mark Garrett led a somewhat controversial discussion on the difference between Internet and telephony. Sincoskie felt the difference was not one of technology but of growth. We are still building half the Internet every year. There are new ISPs every day, many who know relatively little about the Internet and the potential far-reaching effects of a local misconfiguration. (David Conrad mentioned in contrast the stringent licensing requirements in Singapore, at $8M per license, with QoS requirements on availability that grew even more stringent over time.)

In the U.S., at least, the ISP business agenda is typically to sell as many connections as they can for an unmeasured, unregulated and unquantified service. No engineer would claim that either this business agenda or the underlying technology is mature enough to support exponential growth much longer. Internet issues that are several years old are still not resolved: address space growth, routing table growth, routing convergence in big clouds, unfriendly or broken TCP implementations abusing the network. (Sometimes we forget how phenomenal it is that it works as well as it does, a testimony to its architects who were chartered to design a system to handle much less. Moment of silence in homage.)

While we were on the topic of unsustainable growth, Roger Bohn, an economist at UCSD presented his view that a huge part of the problem is that, along with router technology and engineering methodology, the economic model of the internet has not grown either.

The fundamental goal of any economic system is to maximize the number of happy users and the value of what they do. Maximizing value in the Internet is difficult since the economic value model is quite randomized. In most markets, value is attributed according to quality of service. The Internet is no exception: rational pricing would provide the right feedback to providers and users to encourage more appropriate use. Quality signals are not now clear: users need operating signals accompanied by measurement-distinguishable service qualities so they can declare the QoS for which they will pay. Otherwise high value users may get degraded by high requirement, low value users. Regarding the earlier reference to the Internet as a biological system, pricing would form an essential `biofeedback' mechanism to keep the system balanced and effective.

What are the implications for traffic measurement? Measurement at boundaries is essential, and different providers will no doubt measure different things, e.g., source vs. destination-based billing. Furthermore, sample-based billing is inevitable; users need to accept it as just another revenue stream, not precise or optimal. Large customers who care about predictability of service will naturally pay more.

Many ISPs were opposed to usage-based pricing at all, and sample-based pricing in particular. Doran asked what he was supposed to do about customer disputes of charges, and which end of a TCP connection should be charged? Halpern suggested different pricing models for different problems. For best effort traffic, the value of usage pricing is questionable, but for RSVP-based traffic, it makes sense to use the reservation request information for usage-based billing purposes. He also noted that costs for adding capacity are not linear and such an assumption in the pricing model would be incorrect. Ferguson reminded us that ISPs spend more on salaries than on bandwidth; bandwidth is actually relatively cheap. Conrad noted that bandwidth may not be the showstopping cost in the U.S., but it is in quite a few places. Donald Neal from New Zealand (where a usage-based pricing infrastructure has been in place for some time) agreed that a single pricing model would not work.

Sincoskie noted that we are still missing a way to measure service quality. Also, why do we assume the network is congested? AT&T engineered their telephone network for zero loss. Bohn acknowledged that there were two possibilities: either we will expand bandwidth ahead of offered load, or traffic will expand to consume available bandwidth. Either way, cost recovery for resource consumption is imperative to maintaining the `Internet supply'. In a minimalist pricing proposal, we can have both usage and flat pricing, using precedence-based queueing. Users select their own precedence values, and ISPs set quotas or prices for their customers. ISPs count packets at user-ISP boundaries. Users who don't like their service performance can set a higher precedence to improve it; users who don't want to pay for a high quota can set low precedence and tolerate delays. Although precedence is an end-to-end matter, incremental deployment of precedence distinction and quotas can still provide benefits to those who use it.

Luker encouraged consideration of a wider range of existing operational economic models: CoD, US Mail, telephone system's standard, 800, and 900 numbers. Meritorious applications would congest almost any network; Moskowitz's ANX and the education community will need more reliability. Are we going to build the ability to handle these needs into the Internet of the future or have others build it from the outside? Partridge noted that the IETF integrated services working group is already planning along these lines.

For the former category of workload statistics, needed among other places deep within ISP infrastructures, engineers and researchers will have to make a written, clear case to submit to the business side (ISPs) on what data collection would do for them, e.g., solve connectivity problems, allow optimal router design. Ferguson agrees that neither category of statistics provides any current benefit to ISPs; in his own experience as an ISP engineer, he would not have had much use for either workload characterization or RAS. He always knew when something was screwed up and what he needed to do next, before customers complained. RAS would just be a methodology to inform everyone what the operators already knew. What operators need is a model that could, even roughly, predict the immediate next step. Partan agreed: his ISP does not need statistics, they need OC-3 and OC-12 routers to improve service now. And they need them more reliable than the current lower speed routers. Current routing technology just is not built for high performance or robustness, and it will take years to change the industry to support what Moskowitz's agenda presumes. Mathis put it concisely: Internet is like a freight train roaring along while people are laying tracks in front of it. It's not just gaining on those laying tracks; it's gaining on the steel mills.. Fred Baker (at a steel mill) agrees, and noted that the router vendors are a great consumer of any realistic traffic traces so that they can design equipment that will better serve ISPs.

Carter Bullard of Fore Systems presented briefly on security as it relates to measurement. Measuring for security has a more stringent set of requirements than measurement for performance and involves measuring every single network event, and with a high degree of semantic preservation, yet without compromising user privacy. The latter constraint limits investigations to at absolute most the transport layer.

He described Argus, one measurement tool of interest developed at CMU. Experience with argus and other tools have indicated the need for more identifiers in flow measurements, e.g., MAC address; bi-directional. The bidirectional nature of flows is particularly important to capture, for example, whether a TCP connection setup was successful, or if UDP requests were serviced. Measurements also need to comply with theoretical security monitor models, which suggest the need for a physically independent monitor as opposed to an integrated monitor in a firewall or a router.

Security relevant measurements can also be used to support real-time firewall policy enforcement validation, traditional security features such as non-repudiation, and research in the area of analysis of attack strategies.

Claffy presented a more general matrix of the problems we need statistics to solve, created last night with Matt and Bilal Chinoy.

problems we need statistics to solve

need	why	measurements needed	where	problems
trace-driven experiments	cache management optimize queueing congestion and scaling dynamics aggregate transport behavior	full header traces	few high aggregation points (near NAPs)	great political aversion (if becomes part of NAP agreement, will just increase private exchanges) switches can't sniff, need data on all ports or inconclusive
specifications for ISPs to give routers vendors	router benchmarking	flow counts/parameters	NAPs backbone core points corporate campuses large customers private interconnects	software/hardware doesn't support
aggregate traffic flow	capacity/topology planning	traffic matrices	ISPs infrastructures	routers don't support privacy issues
specifications for users to give ISPs	service quality assessments ISP shopping	multi-function transponder testing platforms Mathis' treno Paxson's probe daemon	bridged at strategic interconnects uncongested customers	no validated standard metrics
routing system stability	RAS	route (BGP) logging archiving caching	route servers	most ISPs not using RA (so not great routing data)

and more general needs for

• test points and colocation space
• ongoing dialogue between networking researchers and operational data collectors
• transponder platform most useful if standardized
• network service procurement language for customers
• protocol and application developers cognizant of the mutual effects of their traffic and the operating environment

Bob Moskowitz asked if there were a firm proposal on the table for probe machines and people to which he could contribute resources. He wanted to see 50 probe machines and 3 people to start immediately. Yakov Rekhter suggested starting with `RAS machines' at the three NAPs. Elise Gerich of Merit announced that they were already planning to deploy Vern Paxson's npd software at the route servers, and volunteered Merit to do the post-processing, with advice from Paxson on extracting meaningful information from the data.

Ferguson voiced his concern that we were still not addressing longer term technology issues that have already been ignored for several years. He could not fathom what fulfillment we will get from putting a bunch of probe machines out there and analyzing a bunch of data to come to the solid conclusion: ``Yep, performance really sucks alright.'' What do we do once we've discovered the problems? Where's the engineering methodology that the measurement is supposed to support? Where's the router technology to support the growth we already sort of measure now? How are these measurements going to help us when the router industry is way more than 2 years away from developing the OC-192c routers that ISPs will need in 2 years?