ISMA Survey:
Commercial Sector

Ralph Beck, Net2Net
John Hanley, Yahoo
Brian Kenner, InterVU
John Leong, Inverse
Greg Minshall, Ipsilon
Vaananen Pasi, Nokia
John Quarterman, MIDS
Mike Roberts,
Karen Sage, Cisco
Jeff Sedayao and Cindy Bickerstaff, Intel
Stephen Stibler, IBM

From: beck@net2net.com (Ralph L. Beck)
Date: Wed, 23 Apr 1997 18:06:55 -0400

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

First, to compete and provide a competitive advantage for themselves in the marketplace of access providers. ISPs must acquire the tools necessary to measure network utilization and traffic patterns to ensure that they are providing their customers with appropriate and expected levels of service.

Second, to provide timely network troubleshooting tools when customers are not experiencing 100% network availability. ISPs by definition must provide constant network availability. Therefore, the tools they choose should be easy to use and provide immediate information as to the cause of a problem in the network. Ideally, before any serious impact has been felt by the end user (proactive monitoring). Any significant network downtime will result in lost revenue.

Third, to check WAN links for: compliance by carrier to contract, billing matches usage (checking billing versus actual usage may yield savings greater then cost of monitoring equipment), compliance by ISP to contract.

What type of investments would reap maximum value for the ISPs? for end users?

Hmm, being from a broadband analysis and management equipment vendor this is what I was hoping to discover from the ISPs participating in this workshop!

My perspective, the Internet as a whole and ISPs in particular cannot survive long term based on a flat rate pricing model. ISPs need methods to charge users for consumption of network bandwidth. End users and corporations in particular would be willing to pay more for the consumption of network bandwidth iff they could be guaranteed performance levels requested. ISPs should be looking for equipment that can help them attain this goal in terms of network implementation, consumption billing, monitoring, and troubleshooting.

Facilitate the collection, aggregation, and dissemination of traffic loads, performance levels and network outage data throughout the entire Internet. Privacy and business competitive practices are major roadblocks in facilitating this collection. Develop standard measurement reporting methods that allow cooperation and sharing of network traffic data while allowing the ISPs to compete with each other. Best of luck!

In developing tools, keep in mind end users care about end-to-end performance levels and why they cannot reach a given network resource. Many useful tools are available today (traceroute, ping, ttcp, dig, etc.). However, the interpretation of results from these tools remains up to the user (irregardless of whether they are novice or experienced users).

Monitor end-to-end performance levels and pushback on their providers when service levels are not meeting expectations.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Getting information related to end-to-end traffic measurement and analysis. Correlation of unacceptable end user performance levels with infrastructure problems (physical layer, protocol, equipment, network congestion, etc.) throughout the Internet regardless of whether the problems are transient or long term. You need the facts to know where the problem lays.

A number of good platforms exist on the market to help users with this challenge. Network probes are able to feed information to software applications which track network trends and provide comprehensive performance, fault and capacity planning management. The Internet community should continue to drive the definition of and continued improvements to these products via standards setting bodies and by working directly with product vendors.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

This should be left up to the individuals ISPs. Market dynamics will one day demand that ISPs provide public access to some level of performance and troubleshooting data in order to compete. Survival of the fittest.

The community as a whole should develop a common methodology and format for collecting and reporting aggregated performance data. There is nothing worse than expending resources to provide data conversion that could be better used to analyze the data.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

My somewhat biased response...

Net2Net CellBlaster, other remote management and analysis products.

Long term network monitoring tools: Concord Communications Tracker, Stanford Telecom NetCoach, Visual Networks Uptime.

Visit Net2Net Corporation at http://www.net2net.com

wrt, with respect to
iff, if and only if

Date: Thu, 10 Apr 1997 13:30:27 -0700
From: John Hanley [jh@yahoo.com]

1. In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Any vendor, whether he makes cars, or flips burgers, must keep his product within certain quality parameters to meet the needs of his customers. A process of monitoring the adjusting production is important to this. More rigorous processes are preferrable to ad-hoc approaches.

The ISO 9000/9001 folks know this. Deming did, also. American manufacturers tried to (re-)learn this from Japanese manufacturers in the 80's. The NSP industry has not yet learned this (or has not implemented it).

What type of investments would reap maximum value for the ISPs?

$64k question. I'm not an ISP. I don't know. Maybe financially incenting neighboring networks to treat certain packets carefully.

for end users?

Competitive vendor selection, in near real-time.

The ability to know the current performance of a path to endpoint, the current price of using that path, and the ability to shunt traffic to use the path which customer prefers.

For stability, it may be necessary to shunt traffic "slowly".

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

When buyers know the cost of what they are buying but do not know what they are buying, they cannot make an informed buying decision.

Help me to make an informed buying decision.
Help me to assess what performance I would have seen if I sent traffic via one path or via another.

Since doing this pair-wise for /32's would consume prohibitive resources, some form of aggregation is needed for reporting. Often, a collection of AS's have similar enough characteristics that they could be lumped into a "super-AS". Alas, one might also want to cleave an AS by timezone, or at least into "left-coast"/"right-coast" parts.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Help the market economy to efficiently deliver services that users want.
This will help the network to grow, on a solid economic basis, and will be good for the bottom line of numerous internet players.

Play the PAIX realtime auction game as described by Vixie.
Describe to your NSPs what your business needs to succeed. Hold them to those requirements.
Insist that you know what you are buying (or what you recently bought). Insist that this performance information, once gathered, is made available publicly, not just to current customers, so that prospective customers can make informed buying decisions.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

End2end measurements. The related responsibility/blame/payment issues appear to be infinitely sticky.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

No answer, beyond avoiding the work of processing data if no one looks at it.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Roughly, performance data as an AS-AS matrix, though as I mentioned, ASs which span many timezones may need to be broken up in some way for purposes of data presentation.

The idea is to permit current and future buyers to make informed buying decisions. This incents NSPs to solve user problems, as they will want to convince you that they offer the "best" path to an endpoint. (Most performant path, cheapest path, whatever. Some folks care about loss performance. Some care about latency performance. Some care about effective throughput. Let the marketplace sort it out.)

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

Consumer Reports model.
Multiple reporting organizations which have earned some measure of trust and respect from the NSP and user communities.

The reporting organization may well be the NSP. And/or one or three "Consumers Union" type of designated reporting outfits.
If the publicly available performance data is "good enough", then there is little incentive for additional data collection. We are certainly not there yet. I am concerned that the cloud of a given NSP might be routinely probed by dozens or hundreds of the NSP's customers; that is inefficient, dumb, and doesn't scale.

Intra-cloud performance is relatively easy to report, and is worth focussing on first. End2end is a harder problem; hopefully we won't wind up with "too" many organizations gathering multi-cloud performance data.

Passive collection by end points, perhaps using a few "overhead" fields tacked onto the payload protocol, is also appropriate. The end points might make such data available themselves, or contribute it to clearinghouses, or sell it.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Traceroute is invaluable.
Traceroute.cgi is good, and would be better if it were more widely deployed and better indexed, so that I would have a better chance of being able to gather a reverse traceroute which goes with my outbound traceroute.

Link utilization graphs are hard to come by, and perhaps understandably so.
I don't necessarily need to know specific traffic levels and trends, I only care that utilization was below some threshhold which produces "good" queueing latency and "good" queueing loss rates.

Loss due to router congestion, as opposed to link congestion, is important and is difficult information to come by. SNMP queries by the NSP might update some "congested/non-congested" boolean that traceroute polls, just as NSPs might update such a flag for specific links.

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

Loss should usually be avoided. Not all loss is bad.
RED is a good thing. Knowing where RED and similar algorithms were deployed would be helpful. Perhaps DNS TXT records, which ``traceroute'' looks for, could shed additional light on specific links.

Debugging object code without a symbolic debugger is a poor use of time.
Debugging a cloud without the ability to see some currently hidden variables is also a poor use of man hours. There are folks other than an NSPs employees who have a business need for some of those variables to be exposed.
If the Internet helps net customers to successfully execute their respective business plans, and the Internet and related commerce grow at a rapid rate, that is good for many players, including NSPs.
If you attempt to operate a motor vehicle blind folded, you will have difficulty detecting the detour signs in an efficient fashion.

Date: Wed, 23 Apr 1997 18:01:22 -0700
From: bkenner@intervu.net (Brian Kenner)

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

The Internet is a community of vendors and users. No service provider can quantify its own performance and expect the user to understand that there are other factors in the overall performance equation. Users view and judge the Internet as an aggregate system, not a network of independent systems. Global Internet traffic measurement and analyses allow infrastructure vendors to qualitatively understand and address the aggregate behavior of the Internet much as the audience of end users would perceive it.

Investments in tools and methods for modeling and understanding emerging internet performance data would have the most immediate impact. I believe a starting point for Internet performance and infrastructure problem solving must be quickly established. Examining existing data allows one to focus future development and data acquisition activities based on real knowledge of user problems.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

My hunch is that present communication protocols (i.e., congestion control mechanisms, etc.) are somewhat antiquated_given the types of data most content providers wish to distribute. I'd be interested in determining which and whether these protocols compound the very problem each is designed to fix.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Narrow and prioritize the development and testing process by determining the potential direction and focus of content distribution requirements.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Hmm _ This is a difficult question. I think what's needed is a starting point _ you know, a general hypothesis for the problem(s). A simulation would be nice but has to be based on a fair understanding of the behavior of the aggregate system. WAN measurement is interesting, but is potentially too focused. I believe measurement across ISP's has the most relevance towards characterizing aggregate system behavior. Use this data to develop a starting point.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Acquiring data: Use of multiple methodologies from various perspectives, i.e.
application developers, ISP's, researchers
Storing: Standardized approach
Analyzing: Consensus on priorities and methodologies

Raw Data: Need to know basis only
Raw Results: To those capable of analyzing data and refining conclusions Conclusions: Proposed & final conclusions: General Public via participating organizations

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

One approach, endorsed by majority of providers and beneficiaries

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

We use our tool (InterVU Optimization utility) to encourage the aggregate network to perform better for both our customers (content providers) and their customers, the end user. InterVU approaches the behavior of the Internet from a macroscopic perspective. This view allows us to make the best use of an ever evolving system for our customers.

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

I believe that most Internet community participants are necessarily focused on improving what they do for their own entities. Whatever one does in individual data collection, management, and analysis, the community of ISMA participants must feel that at the end of the day, each has a more complete set of tools to make whatever they do for their business or customer better as a result of their involvement in this industry-wide effort.

Date: Tue, 15 Apr 1997 10:15:46 -0700
From: leong@inversenet.com (John Leong)

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

It is relatively easy to build a network but considerably more difficult to build one that is *managable* ... and it is very difficult to manage a network without data or more specifically, information on the traffic being carried. Tools for data collection and analysis are absolutely critical for network operation and planning.

What type of investments would reap maximum value for the ISPs? for end users?

For ISP : tools for trouble shooting and tools to help prioritize where to spend their next dollar (function of marketing and traffic engineering).

For [desktop] End Users : I really don't want a tool ... I just want things to work. Given that that is not reality, I would like a tool that will automatically detect problems and automatically fix it ... or at least tell me how to fix the problem. That too may be stretching it ... so, given the complex nature of the beast, at least a "Blame-O-Meter" to tell me who to blame when I am having bad services.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

The tools and infrastructure should be widely deployed for maximum effectiveness. However, I would also not underestimate the effort required to design for and operate such large scale measurement infrastructure particularly if they are to be used to support operational rather than just planning need of an ISP.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Given that it is often much easier to trouble shoot if we have data from end points (particularly those people who are observing the problem) over and above those gathered through general observation of the network as a whole, it will be highly desirable to also have tools in the hands of significant end points.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

The bottom line is that end-to-end traffic has to work well. When that does not, the challenge is to find out (a) where the problem lies (Blame-O-Meter), (b) the cause and (c), probably the easiest once (a) and (b) are know, fix the problem. BTW: I believe the single largest performance killer is packet lost given TCP retry timers are in units of seconds while latencies are typically in ms. It does not take many lost packets to make performance unacceptable particulalry for interactive applications [4 seconds here, 4 seconds there ... and we are soon talking in minutes]. This is likely to have very similar effect to cache misses in computing performance.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Privacy is an issue ... but that can mean different things for ISP's, corporations and end users.

The public domain portion of the data should be easily accessible by a wide group of audiences so that they can do their own additional analysis. Given that different people have different analysis programs and tools, it would be very useful if the data format is well published [and easy to handle].

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

See privacy in (a). There is the practical aspect of how much data to keep and how to distribute them particulalry if we are talking about lots of historical data.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

That depends on the domain of the "infrastructure". I suspect the whole thing will take on a hierachical structure both for scalability as well as for administrative and operational control.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

>From end to end point of view, the usual suspects such as traceroute, [and yes ...] ping, ttcp etc. Residue from Cisco router's Netflow cache can also be a gold mine for traffic profiling.

Date: Tue, 08 Apr 1997 21:04:39 -0700
From: Greg Minshall [minshall@ipsilon.com]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

I'm not from an ISP, but i assume to try to detect (and, even better, predict!) bottlenecks in order to shift resources/investments to provide better service to their users.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

I suspect that some researchers may get access to ISP trace data in order to solve some specific problem *for the ISP*. (This might be to figure out why performance is bad at some point in time, or to develop tools for traffic engineering, or ...)

Traditionally, however, we've been better at developing tools to solve *our own* problems. To the extent it is the case that the "campus" (our current locus of operational responsibility/control/access) no longer is a reasonable model of the "global internet", the research community is likely to have less and less to offer to those operating this "global internet". Internet-II, were it to unfold in a certain way, might put a piece of the "global internet" back in control of the academic/research community, which might help.

I think (following on work of Dave Clark, etc.) that end-users should have contracts with their ISPs that provide them certain gross, fuzzily-specified "service guarantees" (things on the order of "x Mbits/sec with t msecs latency", without specifying the interval over which x or t are measured).

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

"Critical challenges" is a bit hard to totally focus on. However, some problems we have (and are going to continue to have) include:

1. what are the metrics? (IPPM is working on this)
2. can we even *capture* the data when it is running at 1Gbps over a *switched* media?
3. what happens when IPSEC is deployed and we lose port number information?

3. What should the community (ISPs, research, users) be considering wrt...

a. acquiring, storing and analyzing data?
b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?
c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

I suspect that ISPs (as well as data centers, corporations, etc.) are unlikely to think it rational to share raw data with the general public. I think that ISPs will collect and analyze their own data. It is unclear (to me) that there is a need to have an "outside agency" do this (this is probably based on a feeling that if we can figure out the economic models -- again drawing on Dave Clark's work -- we can get the $$$ to the location of the bottleneck, providing the bottleneck upgrades, thus giving the bottleneck incentive to upgrade).

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I have two small programs (tcpdpriv and flstats) which i would be glad to "show off" at the workshop. Tcpdpriv removes "private" (sensitive, confidential) data from a tcpdump trace file, allowing people to "sanitize" the file before sharing it with someone (the general public or a vendor troubleshooting a problem or ...). Flstats computes "flow" statistics from a tcpdump trace file.

From: vaananen@NASBPD01BS.ntc.nokia.com (Vaananen Pasi NTC/Boston)
Date: Thu, 17 Apr 1997 15:10:42 -0400

a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

I am not ISP, but I think that this is necessary for being able to understand the behaviour of the traffic on one's network for the purpose of network engineering, trends analysis and performance metering.

What type of investments would reap maximum value for the ISPs? for end users?

Goal should be that the tools required for statistics and measurement information gathering are as much as possible integrated in the networking equipment. If the external boxes and SW are required, there should be possibility to couple the gathered information on the operational management systems used.

As equipment manufacturer, one reason we are participating on these efforts to find out what is important to user's. Other reason is to gather accurate understanding of the traffic behaviour, which is important for the equipment design.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

The availability of the data of the traffic and routing behaviour and trends of those are critical for the development of new networking equipment based on the "old" standards, but perhaps even more important for the development for the new networking protocols and standards.

Provisioning of this data for use of the equipment manufacturers ultimately should yield to better performing and lower cost equipment, which benefits everyone involved.

Hard to give general answer, but I think that organizations operating the network should take a look on their situation, and assess the need for the additional operational data, if any. If operator thinks that for whatever reason, the additional data on some area is beneficial, they should think of how to gather that data and deploy the necessary SW/HW if the value of information justifies the cost and effort required.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Standardization (or at least documentation) of the methodologies, protocols, data formats, etc. I don't see too much technological problems on the gathering data, although some (generally more research, not operationally oriented) analyses are computationally expensive.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

With regards of the raw packet header trace files, some agreement on format would be helpful, as I have made five format conversion programs so far.

Packet traces collected should include both IP and TCP headers, as the interesting information can be generally obtained from those. In ATM environments, cell interarrival times would be interesting in addition to the headert info.

Trace files should be supplemented by the routing tables, as the coarse granularity (network pairs and coarser) flow analyses do not yied to accurate results without correlation to CIDR network masks.

If the privacy is seen problematic, tools should be developed to someway "scramble" the address information *and* routing information without losing the relationships.

FixWest traces have been helpful for the analyses I have made, but the lack of correct address information and/or correlation to CIDR network prefixes limits the usability of those.

The availability of the analyses and tools someone develops is certainly the developer's decision. Generally speaking, raw data should be available for the participating organizations, and it would be nice to have the analysis reports available at least for the participants.

Generally there is no too much benefits on duplication of the effort on the tool development, as it is quite time consuming (only good think for doing exactly the same as someone else is the validation of results...). So please, share your tools and results and tell peoples what you are doing.

It would be good to have some centralized location available to access the information and hold some of the data. But as the nature of the network is quite distributed, it would perhaps be better to do as much as possible of the processing of the raw data near the source to avoid carrying it over network to single location, and doing all the computation on one place.

For this to work, it is of course clear that the standards on formats and what shoud be collected needs to be agreed on.

Above statement is off course true only for the continous operational data, it would be good for the packet trace files, analysis reports, etc. material for the reserach purposes to be available in net site maintained by some organization.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I have used the OC3MON and the heavily modified verision of NLANR flow analysis tools for the flow behaviour analysis. Thank's for peoples involved (especially Kimberly Claffy from NLANR and Joel Apisdorf from MCI) in the development of these tools for sharing them with others and their kind answers to my numerous questions.

We try to use commercially available tools as much as possible (to avoid hassles on support, development and maintenance), but unfortunately have found that the tool situation especially for the data analysis is quite poor.

From: "John S. Quarterman" [jsq@mids.org]
Date: Sun, 27 Apr 1997 12:38:19 -0500

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

So they can detect performance problems before they become outages or customer complaints.

So they can plan future capacity deployments more rationally.

For competitive marketing vs. other ISPs.

For clarifying which performance problems are not within the ISP's network, rather with servers or end-user modems or elsewhere.

For addressing the general Internet perception problem of being slow and undependable by collecting and displaying long-term data, such as http://www.mids.org/weather/pingstats/longterm/htmldir/mean.html.

The first two items above may lead directly to increased revenue by reducing resources spent fixing problems after they become critical. The other three items can lead to future revenue by bringing in more users.

Just as TCP has to handle error detection end-to-end, Internet performance measurements also have to be end-to-end, i.e., from the user to the server, otherwise not all sources of performance problems will be detected.

This is not to say that other types of measurements aren't useful; rather, they're partial.

Instrumenting routers is useful, but is not a complete solution.

Funding.

There is plenty of work for everyone. We choose to measure and visualize performance of the entire Internet, rather than just individual ISPs, WANs, etc.

Multiple approaches. Let natural selection choose the ones that work. Academics, ISPs, end users, content providers, and independent entities such as MIDS all have roles to play.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

The simpler the measurement tool, the better.

For simple measurements combined with complex visualization, see http://www.mids.org/weather/.

Date: Mon, 28 Apr 1997 13:33:18 -0400
From: Michael Roberts [roberts@alum.mit.edu]

1a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis? What type of investments would reap maximum value for the ISPs? for end users?

Measurement and analysis is important because it allows IPSs to really understand the quality of service that they are providing to customers. ISP "products" are likely to evolve along classical lines:

I. at first, basic functionality differentiates offerings
II. as functionality becomes ubiquitous, performance issues predominate

Eventually, best way for ISPs to differentiate their offerings (and thus maintain their margins) will be to guarantee performance. This means they will have to really understand all issues associate with performance (and perceived performance).

The basic goal should be developing methods that allow predictive modeling. Models need to be developed that can both mirror current performance measurements and can say intelligent things about network behavior given changes to the infrastructure and traffic sources. The primary problems here revolve around the dynamic nature of the internet (changing infrastructure), the traffic (emerging applications and proliferation of internet access), the dynamics inherent to modern systems architectures and protocols (interaction between protocol stack layers, adaptive routing and traffic management techniques, etc). The complexity associated with these issues demands a set of predictive techniques that are open and extensible so that they can evolve to match the changing state of the art in technology.

The end-users need to (and no doubt will) continue to pressure service providers to provide verifiable performance measurements that have a real correlation to their applications performance. Information providers have a stake in that they are typically dependent on service providers who can have a large (negative) impact on the perceived quality of their offerings. Information producers and consumers need to continue to develop associations who can represent their common interests to the R&D community.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

The most significant problems related to simulation have to do with meshing techniques appropriate for large network analysis with those necessary to understand very detailed dynamic issues. The fact is that modern networks are both very large and very dynamic. Classical analytical techniques are not capable of modeling performance dynamics. Even addition of heuristics developed through analysis of existing systems can not anticipate the impact of radically new applications and network control mechanisms. Detailed simulation, on the other hand, while capturing these nuances, does not scale well due to the computational complexity associated with modeling large distributed systems. New hybrids of these techniques will need to be developed to allow accurate and cost effective predictive modeling.

3a. What should the community (ISPs, research, users) be considering wrt acquiring, storing and analyzing data?

A common understanding of performance metrics needs to be established. The natural place to measure these tends to be the interfaces between the elements of the system (man-machine interface, client network to ISP network interface). Transmission layer protocols dictate that these interfaces need to be considered wrt their bi-directional characteristics. Without a common set of performance metric definitions, disparate groups can not begin to establish meaningful understandings of their differences.

3b. What should the community (ISPs, research, users) be considering wrt level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

While individual studies will need to incorporate and analyze great volumes of low level performance data, the cost of storing and distributing this data can be prohibitive. It is critical for verification purposes, however, the raw data associated with a study be available for scrutiny by the community, especially since others may have other tools that may amplify or counter a study's conclusions based on a complimentary mode of analysis. For the merits of a study to be debated publicly, all data associated with its generation need to be available, a burden that needs to be borne the producer of the data.

3.c What should the community (ISPs, research, users) be considering wrtc. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

The ISPs should take the lead on developing groups who are chartered to study distributed performance issues. Information producers and consumers should be represented, but it is the ISPs who have the most directly at stake. They are also the organizations who will be best able to identify and gather distributed performance data, at least across the internet cloud.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Comparisons of end-to-end delays for a transaction at each of the various protocol layers involved are particularly useful in determining where performance may be degrading (e.g. application message delay, underlying TCP segment delays, IP datagram delays, AAL PDU delays, ATM cell delays; all measured on the same transaction). Statistical distributions of any single layer performance can be useful in identifying operating and failure modes at that layer or lower. Many insights can also be gained through detailed modeling of a few end-to-end applications (and the intervening systems) in which cross-traffic is based on statistical distributions. This technique can produce accurate sensitivity analyses without the computational complexity of brute-force simulation of many network elements.

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

Issues affecting internet performance will continue to evolve in complexity as the systems, protocols, and applications grow in complexity. Producers, consumers, and transporters of information are making staggering investments in technology, sometimes based on very little hard information about which investments are likely to provide the greatest return. Insufficient tools for measuring performance mean returns on these investments often can not be quantified even after the fact. New tools are necessary to enable all parties to come to common understandings of performance issues and enable them to make predictions in this extremely dynamic arena. The mechanisms for gathering actaul data and for generating modeled data must be open to inspection and extension in order to allow the tools to be adapted to new scenarios.

Network simulation can be a valuable tool in gaining understanding of performance issues. MIL 3, Incorporated provides the OPNET family of simulation tools. Details are available at: www.mil3.com.

Date: Wed, 16 Apr 1997 15:16:36 -0700
From: "Karen M. Sage" [ksage@cisco.com]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Traffic and measurement tools are an investment in people, equipment, and financial resources. The time spent on learning, deploying, and operating such tools and their cost both in dollars and bandwidth overhead are quickly and greatly overcome by the value they provide. Value is measurable through time saved trouble shooting and fixing problems and customer satisfaction in the provision of reliable services. Cost savings are also evident through realizing the most from existing equipment and through effective resource planing and deployment to meet upcoming requirements.

What type of investments would reap maximum value for the ISPs? for end users?

For ISPs, tools that allow the ISPs to control the impact external ISPs have on their own network performance and allow ISPs to charge users (both end users and other ISPs) for use of their transport services.

For End Users, tools which allow auditing of paid services from ISPs to ensure that level of service is provided. Furthermore, tools which help them to control and make effective their own use of ISP services.

Is a measurement infrastructure across the Internet needed? This is a different question then "do you think research into the study of traffic and scalability issues is in the general interest of the Internet" I am not sure which is being asked. I would seriously question the first interpretation of the question, but in general feel research into the areas of measurement and analysis are important.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Understanding the limitations AND benefits of each type of tool.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Anything that is in their own interest.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

The decision should be left up to each ISP/User.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

Why?

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I'm biased but check out http://www.netsystech.com. These tools will be demo-ed during the demo section of ISMA.

From: sedayao@argus.intel.com (Jeffrey C. Sedayao)
Date: Thu, 17 Apr 97 10:25:05 PDT
Cc: cbickers@td2cad.intel.com

Here is our (Cindy Bickerstaff and Jeff Sedayao) statement.

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

ISPs should invest in a process control system of regular data collection, analysis and appropriate action that will detect and remove assignable causes of variation BEFORE customers experience them as problems. This has clear benefits of improved customer satisfaction, attraction, and retention.

In addition, the implementation of a process control system provides the following competitive advantages to an ISP:

1. Shorter MTTR, higher availability, longer MTBF - a quality process control system facilitates the discovery of the cause of departure from expected performance. Lower MTTR means higher availability. Detecting and repairing a problem before failure gives longer MTBF. All of these keep traffic flowing over the network in accordance with designed and planned infrastructure.
2. Smoother, faster implementation of new technology - an existing process control system directs improvement in the new technology implementation process, starts the new technology off with quality controls and enables quantitative decisions about further deployment of a new technology before it damages the existing, revenue generating network.
3. If ISPs do not provide quality control on performance, customers will do it for themselves. In-band, active measurements are a non-value added load on the network. If done by large numbers of customers, the measurements will degrade the very infrastructure that they are trying to measure. Additionally, customers will interpret their data with less network visibility and understanding than the ISPs would be able to provide. This will increase the work load of the ISPs in dispelling customer mis-interpretation of data.

What type of investments would reap maximum value for the ISPs? for end users?

An ISP would realize maximum value by investing in a process control system setup including metric determination (type, placement, sampling frequency), databasing, performance departure detection with alert automation, action scripting, and SPC training appropriate to all levels of the ISP sustaining and development organization (including management).

An end user investment from an end user view would realize maximum value from doing the same with a focus on:

1. Vendor selection and contract negotiations,

2. New vendor, service or technology implementation

3. Planned measurement reduction as the new ISP, service or technology demonstrated quality Internet performance.

An ISP investment to realize maximum value for end users would lie in providing on-line tools such as ping, trace, TRENO, et.al. located at sufficient landmark locations. These tools should allow end users ad hoc sampling capability.

The establishment of a process control system that includes data collection, algorithmic analysis and action scripting is a MUST. Less powerful or "old" tools can be used to develop and debug the system in preparation for the new tools that are developed. If this system is not established first (or in parallel) the value and power of any new tools is significantly reduced or eliminated. Having an existing process control system provides a place to insert new tools which can immediately take advantage of their power (or illustrate their lack of power).

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

End-users can provide regular feedback and review on the tools' goals, purposes, uses. Content providers (in particular, for profit organizations) can modulate the research orientation with some business orientation.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

The most critical challenge in traffic measurement and analysis is less technical than political. The Internet is in a unique position relative to historical process control methods because the Internet consists of interdependant but competitive business networks. As such, measurement and analysis across the Internet that can assign root cause to trouble and enable action to be taken is hampered by competing businesses with concerns about data sharing, data visibility and loss of competitive advantage. Most historical quality process control application takes place within a single business organization with (usually) consistent goals. This "interdependant but competitive" challenge must be met in parallel with the development of more robust tools in order for those new tools to meet their full value.

3. What should the community (ISPs, research, users) be considering wrt...

These are the hardest issues to be addressed!

a. acquiring, storing and analyzing data?

How data will be used must be considered before data is gathered. Experience in other complex systems has repeatedly shown that the proper acquisition and storage of data can make the analysis and use of the data either very easy or extremely cumbersome and resource draining. The ISP concerns in confidentiality or impartiality of this data are powerful considerations.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Keep the raw data with the participating organizations availability. Any additional researchers need to make binding agreements regarding appropriate use of the data.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

There may be role models in either SEMATECH (semiconductor manufacturing industry) or medical trials in the pharmaceutical industry. These may provide some insights. Information sharing issues will be *the* biggest challenge to be met in providing powerful, valuable tools to quality control of the Internet.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Timeit: Regularly scheduled HTTP GET timing to a selected set of "landmark" URLs worldwide. Analysis of both the components of the GET (i.e. socket connect time) and the number and types of errors are powerful in detecting and debugging problems important to WWW users.

Imeter: Regularly scheduled ICMP ECHO to a selected set of "landmark" sites worldwide. Even with the reduced priority and filtering issues, the problem detection power of this tool is still evident.

Server/proxy logs: Analysis of the delivery rates of "value-added" use of the Internet by customers provides a non-loading view into problem detection. The data need to be aggregated to have value which can alleviate privacy concerns.

Traceroute: Still good for debugging after trouble detection from the above tools.

From: STIBLER@watson.ibm.com
Date: Fri, 18 Apr 97 20:15:16 EDT

a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

• To better understand existing traffic patterns and trends, in order to better plan for future improvements.
• Also need to understand existing performance if they have any hope of meeting specific performance criteria set by customers.

• Determination of congestion points and analysis of what additional links might reduce congestion at these points.
• Use of caching/mirrors/tunnels for high demand services.
• For many end users, the critical point may well be the "Last Mile" access. Improvements may be needed here before any other work shows real results for these users.

• As a first pass, measurements should provide data on utilization of existing resources. Perhaps accompanied by measure of quality of service provided. This should serve as a starting point for more detailed analysis of problem areas discovered.

• They should attempt to develop and/or use tools to monitor the the performance being obtained (e.g. end-to-end) across the network. They should also be able to distinguish between "good" and "bad" performance. They should then report problems to providers for further investigation/improvement. Smaller users may need to work together to encourage results from providers.

• Determining meaningful measurements and getting them widely deployed. It is interesting to see case studies of small numbers of test points. It would be more meaningful to be able to compare and contrast the same measurements at many places across the internet. Such measurements could be used to evalute attempts to improve performance. The most important thing is to have a common set of tools available to everyone. Being able to contrast the same performance measurement over time is also important to aid in determining trends.
• The measurements themselves should not adversly affect network conditions. Passive monitoring should be preferred to active testing, and active testing should be designed to minimize the impact on the network.

• Making data available for research purposes while protecting personal data/usage patterns.

• I don't think that the "general public" needs access to the raw data, although it should be made available to people working in this area. I think the public could benefit from higher level analysis of the raw data. There should definitely be some method of sharing data amongst people working in this area.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

• I think the most important thing right now is to make the information about existing tools readily available. Thank you for providing the CAIDA Taxonomy.
• The next step is to provide examples of how the tools can be used, including examples of data generated and analysis, and to encourage deployment of "key" tools.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Work of the Realtime Traffic Flow Measurement (RTFM) Working Group of the IETF.

The RTFM working group had defined an architecture for the collection and reporting of network traffic. Bidirectional flows are defined based upon endpoint addresses (e.g. network, transport and/or application layers). Information is then collected for packets matching desired flow definitions. Currently the data consists of the number of packets and bytes transferred in each direction. Work is in progress for collecting extended information which may help to understand network condition/behaviour.

The URL for the RTFM Working Group is: http://www.auckland.ac.nz/net/Internet/rtfm/