ISMA Survey:
Research Community / R&E Networks

Mitch Baltuch, Unidata/UCAR
Hans-Werner Braun, UCSD/NLANR
Nevil Brownlee, Univ. of Auckland (NZ)
Bilal Chinoy, SDSC
Bill Cerveny, ANS
k claffy, UCSD/NLANR/CAIDA
Les Cottrell, DOE/SLAC
Don Endicott, NOSC/AAI
Chris Fair, NCAR
Anja Feldmann, AT&T Research
Ian Graham, U. of Waikato (NZ)
Paul Hyder, NCAR
Farnam Jahanian, U. of Michigan
Jon Kay, CAIDA
Padma Krishnaswamy, Bellcore
Jamshid Mahdavi, PSC/CMU
David Martin, HEP/Fermilab
Matt Mathis, PSC
Kevin Meynell, Terena
Bill Norton, U. of Michigan
Hilarie Orman, DARPA
Teunis Ott, Bellcore
Vern Paxson, LBL
Brian Tierney, LBL
Kitamura Yasuichi, APAN

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Regardless of original intent, the Internet has become an accepted means of communicating large amounts of data that are used in both commercial and operational settings. A large amount of effort and money has been used to make this happen. Along with this, expectations among users regarding internet reliability have increased. If the Internet is to succeed, it must meet these expectations. An effective way to track the performance and to identify problems is to monitor and measure the traffic moving across the Internet.

What type of investments would reap maximum value for the ISPs? for end users?

As Unidata is not an ISP, I will avoid that question. However, from the end user perspective, online information regarding performance and connectivity would be invaluable. This would include information about outages, congested nodes, etc.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

As I am not familiar with the research efforts currently going on, I cannot comment. However, regarding joint efforts, I believe these are essential to the success of such research. So much of the performance information is privately held, with no access outside of a given ISP organization, that it is difficult to come up with meaningful data across the entire Internet. More widely accessible data would be welcome.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Many organizations and companies involved in using the Internet for providing content and mission-critical data delivery have monitoring procedures in place. While most of these monitoring tools are specific to an organizations task or application, problems that are identified often expose problems with the underlying Internet infrastructure. A communications mechanism between the ISP's and such organizations can be of benefit to everyone.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Lack of publicly available performance and traffic information across the Internet. Also, lack of some sort of access to the underlying specifics of the infrastructure.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

We would most like to see access to long-term performance statistics for the general Internet. This would help us to monitor trends important for operations, as well as help us maintain a data distribution topology that is well suited to the underlying Internet topology.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

I don't believe that the general public benefits from raw performance data. Most people looking at that data would not understand it. However, displays, or summaries of such data would be useful. For those organizations cooperating in monitoring and analyzing network performance, raw data would be of great use. It has been Unidata's observation and experience that packet latencies are important in determining the overall performance of any network. Ultimately, what we are concerned with is the reliable, efficient delivery of data across the Internet. Latencies are an excellent measurement of that.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

At Unidata, to monitor the Internet Data Delivery system performance, we have used a combination of site specific monitoring and analysis, as well as centralized data collection and analysis. Both have their place and are needed. Our individual sites monitor connectivity between themselves and other upstream and downstream nodes. In addition, they report product latency data to the Unidata Program Center, where it is stored and analyzed on a network-wide basis. In the case of individual site monitoring, we have requested that each site make its monitoring data available via the Web. Both individual web site data and data that is available on Unidata's web server are used regularly by IDD participants.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

We use two methods and tools for performance monitoring. The first of these is a Perl script called "netcheck". This is a wrapper around ping(1) and traceroute(1) and provides both logging of results, as well as email notification if packet losses to any node is above a threshold value (nominally 20%). Netcheck is used to monitor connectivity between adjacent IDD nodes. The script is run hourly at each site and the output is made available on the Web.

The second method is provided by the LDM software, which is the application that underlies the IDD. One of the processes that makes up the LDM software suite reports data on product delivery. Information includes, for each datastream, number of bytes received, number of products received, average latency and maximum latency. Each figure is reported for a single given hour. This data is emailed once an hour to Unidata, where it is stored and used to generate graphs, charts and reports. In addition, there is an interactive latency reporting applet available on Unidata's web server that can be used to look at latencies based on a user-specified set of selection criteria.

• netcheck Perl script and configuration file:
  http://www .unidata.ucar.edu/funding/idd/netcheck
  http: //www.unidata.ucar.edu/funding/idd/netcheck.conf
• Site netcheck logs:
  http: //www.unidata.ucar.edu/funding/idd/sitelogs.html
• Interactive latency display:
  http://www.unidata.ucar.edu/funding/idd/StatsClient.html
• Other Graphs, charts, maps, reports:
  http://www.unidata.ucar.edu/funding/idd/iddgeneral.html#status

Date: Tue, 22 Apr 1997 08:59:01 -0700
From: Hans-Werner Braun [hwb@orion.teledesic.com]

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Check http://www.nlanr.net/NA

creation of common service models and metric. The biggest problems are non-technical, but administrative and communicating a common language.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

understanding of workload profiles and trends.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

A common misperception is that a useful service metric of an ISP must be equally useful to a network client. This is not necessarely so. Service characteristics to end-users have to be relevant to and understandable by them. E.g., "my router lost less than 1 percent of its traffic" is meaningless to an end user. Or "my measured round trip delay varied by 10 percent over ten minutes for my video application" is not all that useful to service providers.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

communicating coprehensible results of use to both service providers and users.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data? b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations? c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

create a forum to make it happen.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

whatever helps understand network behavior at multiple levels/views, and whatever is communicatable/understandable to people. Too abstract or complex metrics probably have less impact.

From: Nevil Brownlee [n.brownlee@auckland.ac.nz]
Date: Thu, 17 Apr 1997 22:22:46 +1200 (NZT)

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Without traffic measurements it is difficult to properly engineer a network. Knowing exactly where the traffic is flowing - and just how much traffic there is and how it behaves - makes it possible to determine where money should be spent to improve the network overall.

What type of investments would reap maximum value for the ISPs? for end users?

ISPs have a string interest in knowing which other networks their traffic passes through. Users and ISPs would benefit from measurements of response time for common transactions (rather than just packet transit times).

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

Flow measurments can help in understanding what the users actually demand of the network - how many packets does it take to fetch a web page, what effect does network congestion have on the user's perception of the network, and so on.

For the ISPs, measurements of backbone traffic levels could provide data with which to optimise the network, by determining exactly where new, higher-capcity links would be most useful.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

I think these people should attempt to quantify their requirements, so that ISPs can engineer their networks to meet those requirements. This is part of the shift from 'best-effort' service to something better.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Measurements across ISP clouds seems a big challenge today; having a network of co-ordinated measurement stations providing a growing body of network-wide traffic data should provide a valuable resource for onging research into network behaviour.

One project which I'd see as very useful would be providing servers which would convert an IP address into its Autonomous System number. This is hard to do now (unless you're a router processing BGP updates), but would greatly simplify traffic analysis by ASN.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

It's difficult to answer this in a general way. If you're trying to understand network behaviour, packet traces would be nice. If you're an ISP you'd like traffic volume data analysed by ASN, and so on.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Summary statistics (like the old NSFnet country-by-country matrices) seem fine for the general public. For participating organisations, Plots of traffic rates for specified protocols (web, ftp, smtp, ..) showing weekly variation should be useful. As I said above, it all depends what you're trying to do with the data!

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

Running a library / archive will clearly require resources. I don't think you can answer this question without first deciding what you're trying to observe.

One idea which occurs to me is that since ISPs are responsible for what happens within their networks, observing congestion behaviour at NAPs and IXes should offer some insight to the overall state of the Internet. Measures of NAP congestion could include packet loss rate across the NAPs; this should be corelated with traffic volumes through the NAP. A breakdown of this by ASN would be very interesting!

NeTraMet - of course!

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

I'll reiterate my view that without careful traffic measurements it is impossible to properly engineer a network. Furthermore, one needs to have a clear idea of just what service parameters one is trying to optimise - I guess this means developing easily-understood service descriptions (the step forward from simple 'best-effort').

From: bac@serendip.sdsc.edu (Bilal Chinoy)
Date: Thu, 10 Apr 1997 17:05:00 -0700 (PDT)

1. In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

The key word is "invest". For ISPs that wish to remain in business over the long term, it's critical that they do better asset management, which includes network performance monitoring, traffic engineering and capacity planning.

What type of investments would reap maximum value for the ISPs? for end users?

The commercial Internet is in it's nascent stages. I believe one important aspect is preparing for rapid change. This translates to having qualified people and keeping current with technology.

Having both consumers and service-providers understand the nature and limitations of network services offered would go a long way in setting realistic expectations.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

Define metrics that are meaningful for both end-users and ISPs.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Define concrete requirements. For example, clearly state why any special QOS is required for an application.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

The answer depends on your point of view.

For researchers, there will always be a dearth of data (call it the First Law Of Research: the conclusion is, we need more data)
For ISPs, I think a very important area is understanding elastic v/s non-elastic applications. So, flow characterization and ultimately control.
For end-users, understanding the variances in network performance in a simple-to-use way. So, operating point bounds for throughput and latencies which define an acceptable range of network connectivity.

See http://www.ccrc.wustl.edu/~jpgs/report/ucla-csd-930039.html

I particularly like Merit RA tools which show route flapping ASes.

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

Ultimately, we need simple tools which produce simple metrics, which end-users or their representatives (local network administrators) can run or query. Simple things like determining connectivity is quite complicated now. Traceroute tells you (most times) *where*, but not *why*, *how long*, *other alternatives* etc.

From: k claffy [kc@caida.org]
Date: Wed, 16 Apr 1997 11:42:46 -0700 (PDT)

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

to transition from a seat-of-pants approach to engineering to one based on understanding/models of the workload, in order to have better knobs for tuning/upgrading performance of infrastructure.

What type of investments would reap maximum value for the ISPs? for end users?

hard to say at this point, field is too young. point is they have to start somewhere. setting up measurement machines (flow based and end-2-end pathperformance based) would be good first step.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

make sure they talk to ISPs more than has historically been the norm for research <-> industry communication ISPs need to have vital role in research agenda right now because things are moving too fast for the ivory tower stuff to make any real world sense

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

less antagonistic toward 'ill performing ISPs', more supportive than i've seen so far [i.e., less ANX, more Yahoo]

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

from my perspective it's getting ISPs to seriously invest their attention in it, deploy boxes, support researchers in their attempts, etc but i'm somewhat biased. i wish industry would really see it in their best interest to fund the research rather than letting the government take care of them.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

making sure we can make it public while retaining utility without jeopardizing privacy [e.g., preserve cidr-ness of addresses, but mask them]

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

non-disclosure agreements might be utilized with researchers and with h/w and s/w vendors in order to continue their access to select raw traffic data. protections can be put in place to ensure that company confidential information is not available to competitors. some level of aggregated data (and associated analyses) might be made available to the public.

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

for analysis of Internet traffic patterns/behavior, an organization operating on behalf of ISPs is appropriate, see CAIDA at http://www.caida.org.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

OC3mon / CORAL, see /NA/Oc3mon/, as well as Van Jacobson's pathchar

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

See CAIDA's Tool Taxonomy at http://www.caida.org/Caidants/meastools.html

Date: Tue, 22 Apr 1997 14:25:04 -0400
From: Bill Cerveny [cerveny@advanced.org]

1.

a. Traffic measurement is/will be important to ISPs because

- It will allow them to determine where to focus their equipment and other resources
- Having invested in a capable infrastructure, traffic measurement and analysis may allow ISPs to demonstrate their competitive advantage.

b. While some traffic and performance measurement/analysis can take place without the support or assistance of ISPs, their participation will greatly clarify the picture. Placement of measurement infrastructure at exchange points will simplify traffic analysis and potentially make the analysis more meaningful.

c. The "other end-users" should view ISPs as partners in ensuring that the content reaches the target customer efficiently and at the smallest cost. Traffic/performance measurement and analysis is a mechanism for accomplishing this.

The greatest challenge will not be a technical one, it will be in encouraging ISPs and others to cooperate in a manner which allows for meaningful data and does not compromise (or is not "perceived" to compromise) the proprietary aspects of their business.

3.

a. Acquisition, storage and analysis should be done in a manner which does not

- risk revealing end-user proprietary/competitive secrets or information to a competitor.
- infringe on the privacy of individual(s)

b. Participating organizations who are paying for the data infrastructure and measurement systems should have access to raw data and analyses; the participating organization then shares in the responsibility for protecting the interests of the users.

c. In a perfect world, conflict of interest issues need to be identified, and where possible, avoided.

4.

For everything bad that is said about ping, it is still a very simple and useful tool which is available to everyone. One-way delay measurements with dedicated measurement systems will enhance upon the capabilities of ping by taking some of the ambiguity out of asymmetric network paths and reducing the amount of guessing caused by "bouncing" packets off systems whose job is not performance measurement.

Passive measurement of existing flows can and will tell a lot about the nature of the wide area network. However, it will be more difficult to make precise conclusions about the nature of the network with passive measurements than it will be with active measurements.

Date: Tue, 01 Apr 1997 09:53:15 -0800
From: Les Cottrell [cottrell@SLAC.Stanford.EDU]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

To quote the old adage, "you can't manage it, if you can't measure it". In order for the ISPs to provide cost effective service, they need to:
• understand where the problem areas are so they can be pro-active in problem handling, and know where to apply resources to avoid the/bypass the problems in the longer term;
• know what are the typical use patterns, so they can recognize exceptions, and see trends;
• know what the realistic expectations are that they can set (possibly in contracts) for users, without this they will have disatisfied users who will take their business elsewhere and bad-mouth the ISP.

The most useful and cost effective measurement today is probably packet loss. The ISPs should monitor this on a regular basis and make the results available.

For end users, publishing expectations for quality of service, maybe with credits and debits if they are not met. This would also enable users to shop (and pay for) improved services.

End-users need to do their own monitoring for mission-critical networking requirements. They are the only one who know the particular paths etc. that need to be monitored to meet their needs. The results of this monitoring need analysing and digesting into forms that are immediately understandable to users, funding agencies, and ISPs.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Common documented formats, tools for gathering the acquired data (preferably Web based).

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Mainly long term trends and data suitable for setting expectations. For the short term the needs are mainly for trouble shooting, and the users probably have their own tools such as ping and traceroute or some combination of these. Probably each user has a different set of paths they are interested in so it is not possible to provide information on all possible paths all the time just in case. What the user is more interested in maybe known problems on the Internet at the moment. The user would probably discover (using her/his onw tools such as traceroute) where the suspicious areas are, and then would like to be able to burrow downs into what is going on there (e.g. route flaps, open tickets etc.)

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

For the end user the tools have to be trivial to use, install, deploy etc. the most useful today are ping and traceroute due to their universal availability.

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

Any tools need to be widely available, trivial to install (preferably built in as part of the off-the shelf operating system). Client hosts (i.e the hosts at the end of a path being monitored from a monitoring site) are often uninterested in installing even simple software, and also are concerned about security issues or unknown impacts.

From: "Endicott, Donald" [endicott@nosc.mil]
Date: Mon, 31 Mar 1997 16:08:09 -0800

It isn't much, but you might repost my performance workshop presentation and my more recent ACTS ATM Internetwork (AAI) briefing to DARPA, particularly slide 9 which addresses recommendations. Both can be found at:

http://www1.arl.mil/HPCMP/AAI/

Sorry I haven't had time to write down anything more of late. They still pretty much cover my perspective on ATM-ish performance issues.

From: fair@niwot.scd.ucar.EDU (Chris Fair)
Date: Wed, 23 Apr 1997 15:56:19 -0600 (MDT)

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

As with any demand side economic entity, the ISPs provide a service. As a purveyor of a product they must be able to assess and measure the qualtiy of the deployed of the product. Provision of access services while not content based, may have a profound effect on content. ISP's must constantly measure the quality and level of the product they provide. This knowledge will thus enable the ISP to maintain an appropriate level of service in the marketplace.

What type of investments would reap maximum value for the ISPs? for end users?

Those which allow the ISP to accurately measure its resources,asses the demand on those resources, and the prediction of future resource deployment.

They must be broadly deployed at all levels of the Internet, Research and Education, as well as the ISP level.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

ALL users must be sampled with respect to Internet performance and measurement. Those at the leaves as well as the core network.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

End-to-end consistency throughout the network.

The data should be gathered transparently, without impacting the user/customer meeting all consitutional constraints for privacy and first amendment i considerations. Raw data may be stored in a central location where it is accessible to anyone with an appropriate and legitimate interest in it.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

For the data to be of value, it must be accurate and ubiqoutously available to all user communities.

A level of trust must exist between the user communities and the providers. The data would be most valuable if the interested parties (users) were comfortable with its validity and legitimacy.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I have done a great deal of TCP performance analysis using a modified version of the UNIX utility, nettest/nettestd. This has been very useful for tuning TCP window performance across high bandwidth-delay product networks. Its value on lesser performance networks such as the Internet is questionable however.

5.Please provide any additional comments/position statements/URLs that you would like to share with ISMA participants.

NCAR has worked with PSC and others sith tuned FTP clients for TCP performance. The URL is: http://www.scd.ucar.edu/vg/DCSL/LDT/LDT.html

Date: Wed, 23 Apr 1997 15:24:15 -0400 (EDT)
From: Anja Feldmann [anja@research.att.com]

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Currently the design and operation of ISPs are driven more by a fire fighting (or shall I say trial and error) attitude than by a regular design process. Traffic measurements and analysis should provide the the abstractions that drive the design process for the network, its services, its pricing structure, and its evolution.

Just to give one example: right now we don't know the traffic matrix for the Internet. Yet there are many ISPs that are designing backbone networks. This is like shooting in the dark, especially if we consider the self-similar nature and the unpredictable evolution of data traffic (e.g. Web). Data traffic is completely different than phone traffic where the traffic matrices and demands are very well understood.

Traffic measurements are especially important if we consider the rapid growth of the Internet and some of the upcoming challenges such as adding QOS and large scale multicast. In addition once different pricing plans exist the end users are likely to require certain guarantees from the ISP's in return for paying higher prices.

What type of investments would reap maximum value for the ISPs? for end users?

I am not sure I understand this question. One thing so I can say is that without measurements and data analysis it is impossible for an ISP to estimate what kind of investment will reap the maximum value for ISPs. E.g. is it better to upgrade the link capacity or to introduce Web caches.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

In my opinion there are a ways how the research community can help ISP's:

• define a set of common performance metrics
br>
• develop tools and devices that enable traffic measurements e.g. tcpdump, tcpanaly, etc...


• act as an clearance house for aggregate traffic statistics


• develop statistical and operational sound end-to-end measurements approaches


• identify inherent traffic characteristics that is intrinsic in data traffic (e.g. self-similarity)

While some of these can be done independent others require collaboration from the ISPs; e.g. by installing measurement equipment inside the ISPs network or allowing access to traffic measurements.

Content providers are as much responsible to collaborate with the research community as are ISPs. They should have the same interests in performance metrics as their users. Yet to measure these may again require inside measurement equipment and access to data.

What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Assuming that ISPs are instrumented to do data collection (non-trivial:-) the challenges purely within an ISP could include:

• processing of vast amounts of data


• data integrity (quality), security, and anonymity


• reducing it to its intrinsic characteristic


• verifying that the characteristics are intrinsic


• using it to do meaningful *what if* simulations


• correlating different data sources that are collected on different timescales and at different granularities (packet level traces, flow traces, modem call data, network utilization data (via SNMP))

Across ISP clouds: data sharing and aggregation. Here the aggregation needs to remain meaningful while not revealing proprietary information

What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

See previous answer. In addition careful consideration has to be given to where and how to do data measurement and how sampling can be used to reduce the amount of necessary data collection.

To understand the traffic in the Internet different ISPs will have to corporate and share data. While it is unlikely, unrealistic that ISPs will make their raw data available to the general public, I see it as quite feasible that an independent consortium could succeed in gaining access to raw data from several participating organizations for the purpose of computing agreed upon performance metrics. I could imagine that the computed metrics would be available to the general public.

Of course since the instrumentation of the current Internet is rather sparse a first step will be to instrument and understand the traffic of the individual ISPs

An independent consortium.

The tools I have found the most useful so far has been tcpdump, ping, traceroute. I expect pathchar will be very useful as well.

Date: Tue, 29 Apr 1997 22:56:38 +1200
From: Ian Graham [ian@lucy.cs.waikato.ac.nz]

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

>From the point of view of New Zealand, bandwidth into and out of the country (especially into) is limited. We need information that will enable us to make the best use of the limited bandwidth, and to investigate alternative solutions to the bandwidth problem.

For end users - those that lead to a more efficient use of available bandwidth.

I think we need long term analyses of traffic patterns. in particular to investigate such issues as self-similarity. We also require short-term anaylsis to look at web caching requirements, especially for our trans-Pacific links.

Our interests are in traffic flow characterisation. Obtaining suitable summary statistics for traffic flows is a challenge. So is measurement at higher speeds - eg OC12.

It's probably more a competitive than a cooperative situation in NZ

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

our ATM work on http://phoenix.cs.waikato.ac.nz/atm>

Date: Tue, 22 Apr 1997 16:59:02 -0600
From: Paul Hyder [hyder@triceratops.scd.ucar.edu]

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

It is almost impossible to determine where we are headed without capture of where we have been or are. If it is available, peer groups will utilize this data (in addition to reviewing and proposing expansion of it) to plan and predict their own future.

What type of investments would reap maximum value for the ISPs?

As a member of the user community, will refrain on this.

for end users?

Delay, volume, traffic type, and current load data would help select connectivity options and tune networks. (Provisions for short and long term historical information is a primary need. I rarely need to know the state of the internet "as of this second".)

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

Building cooperation.

Some of the data is being collected, few resources are allocated to analyze it from a perspective beyond that of a single ISP.

Some type of common tool set combined with more global data availability would be useful. (It is usually difficult to access the data, once accessed the different formats make aggregate analysis a challenge.)

Research/education need to provide specific definition of desired data, continue to provide research that produces prototype tools, and to take over analysis of the captured data from the research/education perspective.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Much the same as in b.

A primary concern is that this group will not release their concerns, results, or tools. (Or will need to recover large investments with the resulting high cost information and tools ending up out of reach.)

Availability.

Quality.

Format.

How and where to store the massive volumes of data. At what level to aggregate to minimize information loss that would be critical to another group.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Participating in all possible aspects. (This is a major distributed data system.)

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

The general public? Not sure they would care. Some type of synopsis for non-participants should cover them.

Participating organizations should be provided detail and access based on their level of return to the rest of the community. (i.e. There really isn't a need for more data sinks.)

Central tool and format coordination would be a big win. Beyond that the structure is likely to be very diverse, to meet the specific needs of the individual participant communities.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

The set being presented and discussed is a good sample.

Date: Wed, 23 Apr 1997 12:26:32 -0400 (EDT)
From: Farnam Jahanian [farnam@eecs.umich.edu]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

The justification for the ISPs to invest in measurement and analysis is inevitably driven by economic considerations. Complex systems are too costly to build in an ad hoc way. Performance/ reliability measurement and analysis may prove to be essential in several ways:

• more sound network design and traffic management
• perf. prediction of planned changes to networks
• validation of performance/reliability guarantees required by end-users.

What type of investments would reap maximum value for the ISPs? for end users?

The answer to this question probably will change over time. Initially, for the ISPs, the key investment must be in the form of personnel and also commitment and willingness to share data.

For the end-users, the potential benefits are: validation and assessment of the services expected from the ISPs, and the ability to compare various ISPs. (see part c below)

Cooperation and willingness to shared data and results is critical.

End-users' involvement is critical in defining application-level performance and reliability metrics that are meaningful. They can also play a major role in defining representative application requirements.

In short, end-users must play the role of 'smart consumers' ... they must specify the requirements and demand validation.

I wish to distinguish between technical and social challenges.

Technical Challenges:

• What are the appropriate performance and reliability metrics? One of my colleagues, who is a perf. modeling expert, reminded me that we (ISPs, researchers, and vendors) have to decide what we want to learn before sound metrics can be designed. I should acknowledge the ongoing activities in IETF in defining perf. metrics.
• Can we collect a subset of the data flowing through the Internet and be confident that we can construct models for its behavior? Even modelling complex systems that can be directly measured can be difficult; can we sample large complex systems and build a system-wide model for Internet behavior?
• How do we relate the measurements provided by an ISP to the behavior of the rest the system?
• Another key challenge is to feed the 'real' data collected over the Internet into simulation and analytical models to facilitate assessment and prediction.
• Security and access control to potentially sensitive data? Are we one big happy family?
• Data formats and database issues associated with access and retrieval of data.

Social Challenges:
• Share information among groups of participants (ISPs, vendors, end-users, researchers) with potentially very different goals.
• How can ISPs demonstrate tangible economic benefits in participating in this activity? And I don't mean tangible benefits just to satisfy the technical people ... you need management support.
• How to cooperate and share potentially sensitive data in a competitive business environment?

There are many technical problems associated with a large distributed measurement collection body. These include:

• Data formats. Everyone will probably use their own format for collecting and storing their data. These should be well documented, and convertable to a some standard format for comparison and cross-correlation.
• The many problems associated with large, heterogenous, disparate data bases.
• Ability to index and efficiently access the data.
• How do we cross-correlate the data collected in a wide-area network? How can we be sure that two parties are measuring the same data? Time synchronization (distributed consensus) problems are thorny.
• Ability to sign the data and make it accessible to trusted users. The data repositories must be trusted, and need to have mechanisms for data verification (digital signatures?).

This is more of a social problem than a technical problem. We have dilemma. To encourage participation from vendors and ISPs, it is crucial to protect the sensitive data that they share. There is a danger that this data may be used inappropriately. On the other hand, good analysis (and research) is possible when we are able to share data freely and openly.

I can probably write pages and pages on possible organizational structures and strategies. Let me just say that initially, I favor a tightly controlled organizational structure. Forget about 'checks and balances' ... put a small group of trustworthy dictators in place. :)

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Simulation tools such as LBL's ns can play a complementary role. Modeling tools, such as UltraSan, from other communities can also play a complementary role. I am essentially arguing for using real data as input to simulation and modeling tools.

Many other tools have been shown to be effective or look promising including ping, traceroute, pathchar, netperf (HP), ttcp, and recent IPMA tools from Merit (NetNow and ASExplorer).

Date: Thu, 24 Apr 1997 00:02:33 -0500
From: "Jonathan S. Kay" [jkay@cs.utexas.edu] 1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Because they will be unable to keep their customers happy unless they are able to keep their networks and the Internet as a whole problem-free, and they need to be able to detect and understand problems to do that. The level of difficulties seen on the Internet today and lack of understanding of those problems makes it clear that considerable work is required.

A spectrum of investments is required here. ISPs must not only be able to measure their own networks, but they should contribute resources towards efforts that aid the Internet as a whole. The Internet is really the product ISPs sell, and if that product works poorly, ISPs will lose in terms of growth rate (e.g., from corporate accounts unwilling to entrust much to the Internet). This is surely already happening, see item 2.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

We are currently unable to answer such basic questions as what the quality of service is like for most Internet users.

Quantitatively, we do not know whether the Internet is growing worse or better.

A nationwide/global measurement infrastructure would allow us to answer this question.

Communication. The relationship between the measurement community and ESnet (see SLAC page) seems like a good model.

Currently the biggest challenge is solving the "brownouts" problem. Frequent, short failures of Internet service have become the rule rather than the exception during US daytime hours. This seriously reduces Internet quality of service. And the problem is poorly understood. This is exactly the sort of problem measurement efforts exist to solve.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

We should be concentrating on efforts like measurement infrastructure, which will allow us to understand overall Internet performance problems.

Aside from a reiteration of the vast importance of privacy, none. This is something important to keep in mind - one adverse effect of the Web is that HTTP servers make little effort to protect privacy (note that Usenet servers make more effort to balance privacy against measurement). We must not repeat the Web's mistake.

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

No particular comment.

http://www.caida.org/Caidants/meastools.html ...

My opinions on this subject lie in the text at the top.

I will first reiterate the importance of the "brownouts" problem, the distressingly high frequency of short, sharp performance degradations. See item 2.

In addition to what is said above, I would like to contribute a word on ISP service guarantees. It seems as though packet loss is increasingly the primary metric on which ISPs are measured. This is unfortunate, because it puts pressure on ISPs to install "infinite" queues, and may create artificial pressure against RED installation; both these unfortunate results imply unnecessarily high latencies.

In other words, fulfilling a tough ISP contract may result in worse real Internet service for the contract-holder. Use of a user-level workload - an simple HTML query, a la timeit, would be a good start - seems like a much better idea; if packet loss is too great, it will be reflected in that benchmark to a similar degree as packet loss is really important to actual users.

Date: Wed, 23 Apr 1997 17:16:37 -0400
From: kri@bellcore.com (Padma Krishnaswamy)

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Because, unless ISPs invest in traffic measurement and analysis, their networks will likely

a) not be well engineered for current and future capacity
b) not consistently, perhaps ever, yield the QOS desired by customers
c) incur revenue loss in terms of unhappy customers and inefficient engineering

The best investment by ISPs would be towards collaborative research/experimentation to combine their knowledge of engineering operations with complementary expertise in traffic theory/ network design - the research objective being how to specify / measure /correlate performance, from both an end-end and a network-centric perspective.

A good investment by end users would be to fund research/development on end-end based performance specification and measurement and the associated engineering of the underlying nets, to better available knowledge of how to specify and build nets that meet the requisite end-end performance criteria.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet.

As far as research into tool development and measurements infrastructues, incorporate lessons learned from research settings into network equipment design. Specifically, assess research issues in terms of what might be ported to equipment to better instrument nets with.

Work on a non disclosure basis with specific ISPs. See if specific important net conditions/findings can be emulated/reproduced on research test nets, to generalise or experiment to arrive at engineering insights . If yes, execute and draw associable inferences

as far as content providers, cache based on incidences of flow content (if generalizable) and allied network capacity

Network apps designers: provide specifiable/policable network load

Companies with mission-critical requirements- Support/drive effort to specify and monitor end-end performance with none/variable assumptions re provider collaboration in providing measurement points internal to end-end path.

• Challenges wrt WAN measurement- To gather enough experimental data to gauge the optimal granularity of data collection;establish the most useful traffic parameters for performance measurement in real time; use these parameters to derive effective traffic models. Another challenge is to arrive at methods to gather data ancillary to the traffic statistics such as routes in effect whilst particular data points are observed.
• Measurement across clouds: Find out how to utilize end-end techniques to collect information representative of cloud behaviour
• Simulations: specify real life network conditions (many flows; mix of durations, protocol behavior,required qos, equipment specifics) in forms that can be fed into simulations

  use simulation to evaluate new network design to assess if it will hold up to expected traffic conditions

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

focus on arriving at methods that let network providers collect finely grained data once in a while, and get by the rest of the time with measuring more coarse grained parameters

Storage and Analysis - aggregation and parametrisation techniques; data mining

• The public at large: Observe confidentiality clauses for detailed raw data . Generalise aggregation techniques to mask confidential knowledge whilst retaining technical information.
• Participating organizations: observe confidentiality restrictions.

Aim to focus on a) strict end-end strategies (b) strategies inclusive of network provider participations, say with probes at demarc points and equipment so as to figure out if and how these views correlate

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

There is a mix of old and emerging tools that might be preferrable, but to establish such preference, all tools considered need to be substantiated via experiments as to their usefulness. Cases where these test tools fail also need to be understood- this too can only be gauged via test data/experience. So if this data already exists for a specific tool it should be published; if the data isnt there in a disclosable form,open experiments need to be run to assess the tool's applicability including instances in which it is inadequate

@@summarized in responses above@@@

From: Jamshid Mahdavi [mahdavi@psc.edu]
Date: 09 Apr 1997 16:38:36 -0400

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

To improve the quality of the Internet. In most of the current Internet, performance is far below acceptable standards. In order for the Internet to continue to grow and remain a place for global availability of leading edge information applications, the quality must be dramaticly improved. As outlined below, traffic measurement and analysis is critical to these improvements.

It is hard to define what gives maximum benefits. Clearly, however, there no easy way for users and ISPs to see the perspective of the other. Investment in infrastructure which supports the uniform collection of end-to-end performance information, accessible to both providers and end-users will allow users and providers to view the network through the same lens.

For end users, publishing expectations for quality of service, maybe with credits and debits if they are not met. This would also enable users to shop (and pay for) improved services.

Develop the tools, and then deploy them widely. ISP input into tool development would be useful.

A number of these efforts have reached critical mass and should start producing results shortly after the upcoming ISMA meeting. It would be useful to discuss what results are expected from each of these efforts, and provide an opportunity for the ISPs to provide input into the process of collecting these results.

They should be included in these discussions as well. The end-user concerns will decide what performance metrics should be targetted. In addition, end-user concerns may stretch well beyond ISP issues of network quality/architecture and reach into protocol and applications standards issues which should be carried forth to the appropriate standards bodies.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

All of these. End-to-end performance measurement is needed for evaluation and monitoring of network infrastructure. Traffic flow information is needed to study network usage trends and predict future needs. Better simulation capabilities are needed for protocol developers (who wish to simulate interactions in large, complex networks), as well as router vendors and ISPs who (should) wish to explore how new architectures will perform and scale.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Impact of measurements on the network; broad availability of data.

There is currently a flurry of measurement activity going on. Coordination will be required in order to guarantee that the process of measuring the network scales well and doesn't grow to consume huge portions of the available network resources. Overall architecture for each of these areas must be carefully considered, and developers should be cautious not to release tools before they are ready for mass deployment.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

As much data as possible should be generally available. Especially data focussed on end-to-end performance of the network infrastructure. Because the Internet has become a worldwide resource, everyone has a stake in its quality and successfulness. Developers of data collection and analysis schemes should bear in mind the potential for widespread interest in their data, and therefore plan to address both privacy concerns and data distribution needs up front.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

There should not be *any* required organizations to collect, analyze, or manage internet traffic & performance information. In order to obtain "Internet-wide" data, every organization must be readily able to participate in the process, and there must be no bias based on organizational size, location, or nationality. Measurement infrastructure and results should be globally distributed in some self-managing fashion, such that no single organization stands to gain from or control the results.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

The usual suspects, traceroute, treno, mping, bprobe, nettest, ttcp, etc.

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

I ought to include a URL for the NIMI proposal here, but I don't have one yet.

Date: Tue, 22 Apr 1997 12:28:38 -0500 (CDT)
From: "David E. Martin" [dem@hep.net]

1. In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment,$$$s) in traffic measurement and analysis?

Without realiable measures, there is no way to know what improvements are needed. All ISPs do monitoring, but many use complaints from customers as their primary monitoring tool. It is much better to find out ahead of the customer that there is a problem.

Analysis of network performance data is a complex task. Doing such analysis, though, will give a much better picture of the network than back-of-the-envelope calculations about what might help.

Increasingly, customers want more than a "best effort" promise of service quality. As customers rely more on the network to conduct business, they will write performance requirements into contracts.

The best investment would be in a highly scalable and open monitoring structure. ISPs should share performance data and use a common method for data collection and analysis. This would benefit both the ISPs and end-users, by giving them a common frame of reference to discuss network problems and performance.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

I am a member of the research community, so I will answer from my perspective. The tools deployed should be highly scalable. This means they should have little impact on network traffic, ideally none. For a very widely distributed system, passive monitoring is probably the only solution. In the short term, low-impact tools like ping should be used.

Other users should keep the ISPs honest. This may mean having monitoring and analysis efforts of their own, but in the very least it means checking the methodology of collection and analysis of the ISPs and demanding meaningful reports.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

The most cirtical challenges are measuring end-to-end connections crossing many ISPs. Each ISP treats their network as a closed system and the gateways out of it as the edge of their domain. This must change. ISPs must care about the paths between users.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Data acquisition should be done via non-obtrusive means, ideally passively. Data should be stored for long-term analysis and made avaiable to everyone.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Being a researcher, I would like to see raw data and full analysis be available to everyone.

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

Although I would like to think that ISPs will become more responsible in monitoring and making the data available, I believe that end-user communities will be ultimately responsible for data collection and analysis. Communities of interest (i.e. energy research, space research, etc.) will have to provide collection and analysis across a myriad of ISPs.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

The tool I have been using most heavily is ping, wrapped in perl scripts that automate data collection and anaysis. For graphical data analysis, I use SAS/graph.

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

I believe that a system that monitors only a single ISP is useless to the end-user. Even in relatively insulated communities such as high-energy phyiscs research, the majority of traffic is to non-traditional sites. Monitoring and analysis must follow actual usage patterns.

From: Matt Mathis [mathis@psc.edu]
Date: Wed, 23 Apr 1997 10:08:22 -0400

1a) why?

Better measurement is needed to help drive the ongoing deployment of Internet infrastructure. Any measurements which help us to anticipate tomorrow's Internet will strengthen our preparation.

What type?

The Internet is so varied that nearly every measurement technology is useful in some context. We (PSC) have chosen to focus on congestion, and the relationship between phenomena observed in the network (loss, delay etc) and end-to-end transport behavior.

1b)

scaling

1c) Role of Internet users?

The largest role of the Internet users will be to use their wallets to express their priorities. It is our role to help them make informed decisions.

2) what is most critical?

The Internet is so broad and varied that no particular challenge is ubiquitous. We are focused on understanding the end-to-end performance of long paths, and how it relates to charteristics of the composite segments of different providers.

The most critical overall problem is the shortage of skilled staff. In this regard Internet measurement is often in direct competition with the Internet itself. Providers faced with ad-hoc measures of poor performance often have the choice between expending resources to improve the performance or to more accurately measure the poor performance. In the short term, it is most likely to be better to invest in the Internet itself. Some less successful providers may never be in the position to consider the longer term.

3a) ... processing data?

The data should be archived after various degrees of processing, such that people needing the broad view with limited data capacity can extract highly summarized data. People with more date capacity who want more resolution should be able to get it. The R&E community and agencies already own large computational and archiving assets that could be used to process and archive the data. Although most networking people think of our data sets as large, they are minute compared to many existing Internet databases in other fields.

3b) ... level of detail?

It is clear that one key aspect to motivating ISPs to participate is to make access to results contingent on the level of participation. This relationship must be continuous, without large policy discontinuities in either participation or or access.

3c) ... org structures?

Market driven anarchy, with periodic review of all current activities. Simple annual meetings of all people implementing Internet measurements may induce optimal levels of collaboration (Sufficient overlapping methods to be confident that the solutions are good, but not so much that there is outright duplication of effort).

4) Tools?

See the TReno test page at http://www.psc.edu/~pscnoc/treno.html or the paper at http://www.psc.edu/~mathis/htmlpapers/inet96.treno.html

Date: Wed, 23 Apr 1997 15:48:15 +0200
From: Kevin Meynell [K.Meynell@terena.nl]

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

There is a large cost associated with providing equipment and other resources (at least in Europe). ISPs need to ensure their resources are being focused in the right areas. For commercial ISPs, and to an extent research ISPs, traffic measurement and analysis will allow them to demonstrate a competitive advantage.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

It may be difficult to persuade some commercial ISPs to participate in measurement analysis, so traffic measurements probably need to be conducted at Internet Exchange Points.

End-users must encourage ISPs to publish traffic meaurements in order to ensure their data is being delivered efficiently. It also works both ways however, as users are often unsure as to who's responsible for bottlenecks on the network. Traffic measurements would allow ISPs to 'absolve themselves of the blame' for poorly performing parts of the Internet.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

I believe traffic measurements across the WAN and traffic flow information are the most important issues. Measurements across the WAN allow the monitoring of network infrastructure, whilst traffic flow information will help predict future needs. Network simulations are of secondary importance, certainly to the European Research Networks.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Someone has already made the point that data collection should be non-obstrusive. It would be unfortunate if traffic measurement was actually contributing to some the congestion we see on the Internet today.

It would be an ideal if information was made available to the general public, but we must be realistic. In order to encourage organisations to participate, I think information would initially need to be restricted to the participating organisations.

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

Standard tools and methodologies for collecting and analysing data.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I'm unable to comment as I do not (yet) have experience of any specialist tools.

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

TERENA has established a European Task Force to deal with the problems of traffic measurement and analysis. Please see the following URL for more details:

http://www.terena.nl/task-forces/tf-etm/

Date: Thu, 24 Apr 1997 15:40:45 -0400
From: "William B. Norton" [wbn@merit.edu]

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

During the operation of the NSFNET (particularly during the latter stages of exponential growth), performance measurement was critical for capacity planning, adjustment of metrics for load balancing, and for the detection and identification of systemic problems. Besides all the usual technical reasons for measurements, one must consider the fact that the Internet is a *shared* complex system. Actions in one part may have dramatic effects on another. Advanced warnings of material changes in this system would be useful for planning purposes, but instrumenting this rapidly evolving system is clearly important to help identify that these changes have occurred. Without such instrumentation it will be very difficult for the Internet to grow rationally.

What type of investments would reap maximum value for the ISPs? for end users?

Research is a very costly endeavor, so investments in shared expertise, infrastructure, and data leverages the shortage in capable staff and financial resources would maximize the value for the Internet system. But the flip side is that contributors expect value in return. Vague assurances that "its the right thing to do for the community" or "the value won't be seen for some time" pale in comparison to the ROI on additional sales and deployment engineers.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

The difficult hurdle will be policy reasons for not participating. I would recommend making the measurement infrastructure as close to free and no maintenance as possible. I would also suggest adding some direct and immediate benefits to having the device on site (perhaps some display of health of the rest of the Internet beyond the point customer DMZ).

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

These folks all have a vested interest in the stability and rational growth of the 'net. Many of them are already performing their own measurements. We ought to provide to them the tools that give them what they are looking for in their measurements, that by default contribute the "right" measurement results to the community. (Privacy issues will not be addressed in this memo ;-) )

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Standardizing the tools and coming up with the methodology that will be public. Traffic characterization is critical in the sense that the equipment providers and ISPs need to be able to plan for the future, and this is often done with data from the past. Of course this preassumes that more ISPs are a good thing for the infrastructure today.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Models are critical here. Distributed data store versus centralized. I like the distributed models because folks can "own" their own infrastructure for measurement and it scales well. The trick then is to make them tie together seamlessly.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

We certainly need to operate on a higher level than the raw data. The general public shouldn't need to see this stuff. They should see a single number - The Internet Service Provider is rated with 4 stars. The details are of interest to the providers, equipment vendors, and more knowledgeable shoppers. I would expect an aggregation of data to the network level, where we focus more on "trends on critical metrics."

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

Too many obstacles and choices today to answer this. As mentioned earlier, these technical issues are small in comparison to the policy and rational anti-cooperative motives pervasive in parts of this industry.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

A central place to query for network health - like the national weather service.

Date: Mon, 21 Apr 1997 12:11:40 -0400
From: "Hilarie K. Orman" [ho@darpa.mil]

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Users will not rely on a system that cannot be viewed as having predictable performance with contractual guarantees. Today we have only a minimal understanding of what the performance is and what guarantees are possible. The basis measurement and analysis is needed for the foundation of future engineering efforts that will guide the industry.

Investments that tie the measurement to the infrastructure in a uniform and secure manner are important. The most important measurements are those that will lead to improving overall performance and those that can be used to detect and fix problems in real-time.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

It is important for the research community to keep in mine the practical difficulties of measurement in real traffic conditions, the problems of maintaining the software, archiving the data, keeping the tools properly configured, etc. ISP's and equipment vendors should be consulted and involved in these issues. Beyond that, ISP's have insight into what factors most immediately affect their business plans, and the research community must produce tools that are or can be made relevant to business planning, e.g. graduated levels of service, etc.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

The requirements must be fed into the measurement planning, but it is important to realize that user requirements will vary as the industry continues to develop, so the tools must be immune to changes in how the technology is used.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

The most critical issue is to identify anomalous or "dangerous" patterns quickly and when possible, to have hooks for automatic response measures. To this end, an API for interprovider information sharing is crucial.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Tools that are scalable and easy to use, standard interfaces for vendors, ISP's, researchers, privacy, use of modeling to guide the acquisition and analysis, independent organizations for providing statistical services, including use of supercomputers for analysis.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Because of the privacy concerns, we still don't know the answer to exactly what should be available to whom. The most important goal is to have software that is capable of releasing data according to policy decisions, whatever those turn out to be.

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

Interprovider sharing should be facilitated through standard interfaces, and the industry should voluntarily adopt standards for reporting and working with independent experts. Reports should be available over the Internet, and visible in as close to real-time as possible. Government can help guide this process in conjunction with research into the real-time control issues.

In the future consumer groups will demand greater accountability and changes in business practices, and it is important to develop a sound statistical basis for evaluation now.

Date: Tue, 22 Apr 1997 13:44:59 -0400
From: tjo@bellcore.com (Teunis J Ott)

1.In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Two answers:
1. Competitive Advantage: Once ISP becomes a truly competitive business, QoS (used in advertisements, word of mouth, etc.) becomes a decisive advantage. Network engineering will give some (how large ?) efficiency and cost advantage. Example: One of my friends (small business) has accounts with TWO ISPs (for reliability reasons). He would prefer to use only one.


2. If the ISPs do not do it themselves, somebody will do it for them. (Monitoring, and ruthlessly publishing the results). (mostly QoS).

What type of investments would reap maximum value for the ISPs? for end users?

Don't know. I would like to know the answer to this question!

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

This question is too early, too much.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Content Providers would mostly be customers and thus they vote with their wallet: Run away from from ISPs that do not deliver cheap, reliable service to my customers and the customers of my customers. Share their knowledge with their customers on which ISPs are more than / less than reliable.

Mission critical: This is likely to be a real market-push! Companies with mission-critical needs will write special contracts with special tariffs and penalty clauses and will do their own monitoring of reliability / QoS.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

That is quite a mouthfull.
I consider understanding of Source-model behavior critical (more than of traffic flow characterization, which is largely determined by the network, not the customers). Traffic measurements, incl across networks, is also critical. Traffic flow characterization is interesting but second order.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Let's do more of this and philosophize later.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

There are two extremes: Fine grained traces (of packets, cells, etc, with microsecond timestamps) and course-grained measurments (say 5 minute ``totals'', where ``total'' is more than a single number). Both are essential.

c. the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?

That's what we will discuss in this meeting, right?

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Measurements: I have not followed the technology lately. I use math and simulation and try to understand the protocols and customer behavior. Seems to work.

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

See my paper ftp://ftp.bellcore.com/pub/tjo/TCPwindow.ps (contains a little bit of math, still nice).

It seems to me that the drivers for this work currently are:


(i) Service differentiation: creation of special types of services with special needs (reliability, QoS).
(ii) Monitoring, to check whether services with special needs are getting what they agreed to pay for.

The reason these are drivers is that the companies with special needs are the ones with money and incentive.

Vern Paxson

Link to Vern's paper entitled Measurement Infrastructure for the Next Generation Internet

Date: Tue, 15 Apr 1997 21:16:05 -0700
From: Brian Tierney [tierney@george.lbl.gov]

Answers to several of the questions can be found within the following document:

http://www-itg.lbl.gov/ DPSS/logging/

As developers of high-speed network-based distributed services, we often observe unexpectedly low network throughput and/or high latency. The reason for the poor performance is frequently not obvious. The bot- tlenecks can (and have been) in any of the components: the applications, the operating systems, the device drivers, the network adapters on either the sending or receiving host (or both), the network switches and routers, and so on. It is difficult to track down performance problems because of the complex interaction between the many distributed system compo- nents, and the fact that problems in one place may be most apparent somewhere else.

Network performance tools such at "ttcp" and "netperf" are commonly used to determine the throughput between hosts on the network. While these are useful tools to start with, we have observed many cases where ttcp performance is reasonable, but real application performance is still poor. Monitoring status and error data from the network switchss and rout- ers also may not point out any problems. Real distributed applications are complex, bursty, and have more than one connection in and/or out of a given host at one time; tools like ttcp do not adequately simulate these conditions.

The best way to diagnose these types of problems is to run real distrib- uted applications that are instrumented in a way to collect accurate timing information at all important events. To facilitate this, we have developed the NetLogger Toolkit. This toolkit includes code that applications can use to generate event logs, and some simple tools to help graph the results.

1. In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment,$$$s) in traffic measurement and analysis?

Because, that measurement and analysis helps to know the tendency of the activity of the ISPs. That is, the effect will show the weak point of the network or end users' tendencies. And it also showed what point ISPs should invest their budget.

What type of investments would reap maximum value for the ISPs? for end users?

For ISP, to invest some high speed equipments in rebuilding up their backbone would reap maximum value. Because it is possible for ISPs to prepare a lot of services.

For end users, to invest some technical staff in the NOC would reap maximum value. Because those staff will help to recover backbone if there are some troubles at the back bone.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

The Analysis for source/destination address and the service protocol will show the end users' tendency at the network. To know the kind of service and the destination/source address will help the network design. That will show the efficient traffic exchanging way.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Those user's role is to show the limit of the network performance. That is, the process speed of the exchanging equipments, or protocol throughput.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

I view the traffic flow characterization. In these days, a lot of network application are used and sometimes the information for the applications arrive as flood.

3.What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

ISPs and researcher should be.

b. level of detail of raw data and resulting analyzes available to the general public? available to participating organizations?

Only available to participating organizations, i think.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

Yes, the community should be.

4.Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I cannot find such special analysis tools yet.

5.Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

APAN stands for Asia Pacific Advanced Network which consists of
a)Intercontinental Hub System
b)Backbone links connected to the intercontinental hub system
c)other links connected to the intercontinental hub system

At http://www.apan.net, you will get more information about APAN.