ISMA Survey:
Internet Service Providers & Exchange Points

Randy Bush, Verio
Steve Feldman, Worldcom
Henry Kilmer, SprintLink
Rakesh Kushwaha, AT&T
Cleveland Mickles, MCI
Mike O'Dell, UUNet
Curtis Villamizar and Daniel McRobb, ANS
Rick Wilder, MCI/vBNS
Bill Woodcock, Zocalo/PCH

Date: Sat, 19 Apr 97 18:55 PDT
From: (Randy Bush)

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Beacuse circuits are 40-50% of a US ISP's costs and even more for a non-US or multi-national ISP. If I can be more efficient in equal or better service provision than my competition, then I, as engineering, have done my best to give my salesdroids the advantage.

Crass, eh?

What type of investments would reap maximum value for the ISPs? for end users?

I do not understand this question. Are you asking what kinds of measurement seem to have the best payoff today?

Provide the basic metrics and, in their traditional role, the first tools.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Hard call. They will want to measure different providers, but can only measure those to whom they are connected.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

Lack of market understanding of issues. Lack of metrics and tools.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

My guess is data sharing is an issue, so we do not waste a lot of resource on duplicated efforts. In that case, common tools and data format issues will become germane.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

The research community has a way to go in realizing the ISP community's concerns here.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

Academics, industry, ISPs themselves, ... What would be the gain in limiting it? But do note that there are privacy issues when data are shared or otherwise exchanged.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Nothing shockingly useful. Primitive circuit tools do not even handle PVCs and VCs.

Date: Tue, 22 Apr 1997 10:54:07 -0700 (PDT)
From: Steve Feldman [feldman@mfst.com]

1a.

Accurate traffic measurement and analysis can give providers the ability to observe and predict network growth, understand and predict network behavior, and (for those that care) generate accurate bills for usage-based services.

ISPs can help each other by assisting in the definition of metrics and analysis techinques, collection of data which can be made public, and helping the rest of the community understand the correlation between metrics and actual network performance.

1b.

The research and education communities should work together with the ISPs and customers to define measurement and analysis tools and techinques which would be value of value. The R&E community can help by creating and standardizing measurement, analysis, and simulation techniques and tools, coordinating measurement and analysis efforts to avoid duplication of effort, and acting as a clearinghouse for non-proprietary data.

The government itself should not be directly involved in the collection or analysis of Internet traffic data (shades of Big Brother!), but can help by providing funding to the R&E community.

1c.

End users care most about metrics as observed from their own points of view, at the edges of the network. The end users can help by assisting in the definition of these metrics, and providing useful test points on the Internet for measurement.

2.

There are two critical challenges.

First is the ability of network elements (routers, switches, etc.) to accurately measure and report network perfomance data. This includes not just byte and packet counters per port (though even these are often woefully inadequate), but the ability to measure traffic flows between pairs of ports and/or user addresses. The current trend toward embedded RMON technology is a step in the right direction, though care must be taken that measurements can be performed without the measurement process affecting network performance.

Second is the ability to "anonymize" collected metric and packet trace data so that it can be made public without divulging proprietary information. Network providers are reluctant, and sometimes contractually obligated not to, release any data which might compromise the confidentiality of their customers' traffic.

3a.

There should be common formats for data storage, to facilitate data exchange.

3b.

It is critical that there exist methods for ensuring the confidentiality of user data. Both endpoint addresses and packet payloads should be obscured before any data is made public. Standard techinques and data storage formats to facilitate them would be very useful.

Data collected by or released to the R&E community should be available to the entire R&E community. This data should not be available for anyone to use for marketing or promotion purposes.

3c.

The most important organizational strategy is to ensure that this work is done in an efficient manner. Duplicate or incompatible efforts should be discouraged.

5.

We should take great care that this doesn't become "Internet Consumer Reports". There is a place for that sort of activity, but it shouldn't be in the R&E community.

Date: Wed, 16 Apr 1997 22:05:40 -0400 (EDT)
From: Henry Kilmer [hank@rem.com]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

It can enable an ISP to do better traffic management and network design. From a business perspective, this means more revenue. >From a engineering perspective, this means a better performing network.

Traffic engineering mechanisms are severly lacking in today's tools. Cflowd is a step in the right direction by helping generate an AS based matrix in order to understand traffic flow. So from the ISP perspective, more ways and better ways to understand where the traffic is going and why is key to success.

One of the key areas of concern that we are hearing from our customers (ie: end users) is the overall network performance and QoS. Ways to accurately understand and track network performance will become mandatory as QoS becomes more real.

My first recomendation is actively pursue the ISP to work WITH the researcher. This is key since most ISP's are growing extremely quickly and the engineers are very busy. Some ISP's are very willing to work with researchers to better understand the network (both the individual ISP's and the overall Internet) which can help everyone involved. Better coordination between the communities will generate more progress in less time.

My second recomendation is to actively and jointly pursue defining what information is necessary for success and pressure the router and switch vendors to come up with ways of exporting that data without impacting switching performance.

I would like to see the content providers heavily involved in traffic measurements. Their is a great deal to be gained for both the content providers and the ISP's in better understanding their traffic flow and demands and using that information to improve protocols and implementations of products.

Companies with mission-critical networking requirements must understand the performance they are receiving from their provider(s). Period. If they care, the next step after understanding what performance they are receiving is to understand why and that involves working with their ISP(s).

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

In a word, time. WAN's today push the capabilities of the routers to the limit making it difficult or impossible to use the routers/switches for gathering statistics.

Many ISP's have tools that measure their own network but never look beyond their administrative boundary. Today measuring across multiple providers is left to the end users. The problem that I see with this scenario is that in many cases, the end user is blind to the infrastructure which is necessary to accurately interpret many results (for example:traceroute in a closest-exit world confuses many).

There has been a great deal of effort lately put into tools to assist in network simulations. From what I've seen, these *can* help evaluate network state based upon various criteria. The problem I have with the work that I've seen is that it does not reflect real world where my outages rarely seem to be "hard" outages. They also seem to be one step behind with software features or interface speeds than reality. Having those tools catch up and keep up with reality will be a huge challenge.

a. acquiring, storing and analyzing data?

The data has to be able to be stored in a compact but fast means.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

With $$$ comes proprietary information. For some information, non-disclosures will become mandatory. There are still some engineers and companies that understand that sharing this information helps the industry and are willing to do so. So some data will make it out to the general public and some will remain in the hands of the organizations will to join forces and better understand this industry.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

End user data is critical - believe it or not, ISP's do not always know what their customers want. I also believe that core infrastructure data analysis is critical. It is the best way to get realistic input to equipment vendors, protocol designers and implementors to keep the network growing and usable.

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

Cflowd is very useful. For an operational perspective, MRTG with its various tools (like rdlog2) are also very useful. For analysys, ttcp and treno have helped.

From: Rakesh Kushwaha
Date: Thu, 24 Apr 1997 16:11:18 -0400 (EDT)

a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

To better model and engineer their network, and to predict its performance, reliability and capacity for future growth. Moreover, it would also help ISPs to optimally configure their NAPs to peers.

Investments which would allow ISPs to correctly model the network, to predict performance and growth, would reap maximum benefits. So that ISP's can provide desired QOS to the end-user.

For end users, investments which would improve QOS and provide services in timely (fast but deterministic) and reliable manner, would be beneficial.

The methodology of measurement/analysis which research/higher ed communities and government are engaged in should be heavily publicized and advertised, so that similar methodologies could be adopted by ISPs as well. Both, ISPs and research communities should decide on common metrics and exchange analysis results frequently.

Content and service providers should monitor traffic for each and every service they provide, define strict requirements for the network and service performability, track them and devise methods to meet them.

2.What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

WRT WAN Define uniform metrics and design methods to compute them without impacting the network performance.

ISP clouds : Determine in real-time if the delays, packet loss and congestion are on access side or egress side, and how to control them adaptively in real-time.

Flow : categorize traffic flow on the basis of some traffic parameter not on basis of protocols. Determine the effect of one type of flow on the other and predict scalability.

simulations : Simulate actual, present and real time Internet. Traffic situations change fast, simulations should cope up with the changes.

It is very important to archive the data and results, data whare houses must be established to do the same.

Resulting analyses for sure, should be made available to everyone. The source providing the Raw data should decide what needs to be filtered.

The organization taking the responsibility for the Internet traffic data collection and analysis should constitute of performance analysts (individuals of mathematical background), networking experts (individuals with network architecture knowledge), technology consultants (business experts to predict where Internet is heading towards), and service experts (individuals who know how to provide various end-user services). The organization must decide how, when and where the measurements should be taken to get meaningful results.

Since the traffic between clients and servers is asymmetric. Tools which help evaluate and measure one way performance metrics without putting too much load on the network would be particularly useful. Currently, within AT&T we are evaluating packet-pair theory to measure one way congestion and delays.

Divide the work, repetitive work should be avoided.

From: Cleveland K. Mickles cleve@mci.net
Date: Wed, 16 Apr 1997 11:45:40 -0400

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Many ISPs tend to oversubscribe their networks in the hopes that not everyone will try to access it simultaneously. Analysis should continually be done to trend the traffic growth curve to determine when the network will begin to run into capacity related problems.

What type of investments would reap maximum value for the ISPs? for end users?

ISPs should invest in people to design and build a network that will handle current traffic growth trends for at least 12 months. Work with hardware/software manufacturers to continually add statistics gathering capabilities in their products so that we won't overload current harware with SNMP requests. An ISP must have the ability to maintain stats on interfaces throughout the network to determine trends in traffic growth and plan to increase capacity prior to certain thresholds being crossed.

ISPs should provide reporting cabilities for the end users. At a price, of course.

The tools can be deployed in various points on the edges of the various networks being run by ISPs to gather data, although direct access to this data by external sources would probably not be allowed.

Monitoring NAP connections on both the LAN segment as well as the tail circuit back to the ISP would be usefull in identifying how much traffic is being exchanged between two peers at a NAP. That info would be helpfull in determining if a new peering point would need to be established between the two providers involved.

There needs to be aggreement on how the data is presented to the public or ISP community. There would need to be legal documents signed between the party's involved to make sure guidelines are followed. The guidelines should be in affect for a certain period of time.

There are things on the table such as distributing servers at various points in the internet and web caching. I think we would all agree that these are great solutions assuming the content provider's circuits are full but if the problem is network congestion then that again falls into the realm the ISPs ability the provide a reliable service. Content providers have bought a service just like any other internet subscriber and they should push their ISP invest in infrastructure to support traffic requirements while realizing that you get the service you pay for.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

WAN/CORE measurements are usually an aggregation of so many flows( in MCIs ATM CORE network) that not much can be drawn( by external organizations) from those measurements other that how close to capacity are an ISPs trunk circuits running. Analysis of trunk circuits *should* be pretty straight forward and publishing those stats can only provide a negative impact on how close ISPs live on the edge in terms of traffic thru their CORE. That info would only be usefull in justifying the purchase of more capacity which is usually an internal decision. Most of todays hardware provides the basic capabilties necessary to monitor high speed WAN links.

We could consider publishing stats for customer connections which can be aggregated to determine aggregate bandwidth flowing thru a box but we would need to know if a customer wanted this info made public. There may be legal constraints if customers don't want their stats made public.

At some point in the future usage based billing will become widespread and this would allow a direct correlation to a customer bill based on port stats. This info may be used by some analyst in the financial markets to determine how much revenue an ISP is making. Although we have the best intentions today, it's just not clear how the information would be used and how it could affect the perception of an ISP(MCI in particular) in the larger community if we decided to discontinue publishing certain information in the future.

Are there any tools available to monitor multicast traffic?

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Binary data would need to be collected and stored within the ISPs network. The data should not be directly accessible for security reasons. Polling intervals should be tuned such that gathering statistics does not cause degradation in switching packets thru the network.

The analysis of the data should be done in conjunction with the providers. Some ISPs may decide they do not have the resources to perform such analysis and may opt to allow the research community to perform the analysis assuming there is aggreement on how the data is presented.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Generally stats should be made available to customers which shows their utilization of customer connections. With this info and other pieces of information(ie costs) in hand, customers can determine if they need to consider making a change to their current configuration. Or possibly identify who/what regions are accessing their internet site.

Information on the internals of an ISPs backbone would serve no usefull purpose to the public, since the traffic there would consist of numerous aggregated flows. This info could be misinterpreted and may reflect negatively on those providers who oversubscribe their backbone trunk circuits. MCI has taken the position in it's network design that the center of the BIPP network is designed to not congest. This info is used internally to justify adding capacity to the CORE of the network and making this information public would not aid in getting that goal accomplished.

NAP connections on the other hand have been a major source of pain between most of the large ISPs. Usually those links carry a large amount of traffic and we have attempted to compensate by adding more of these connections but the solution to the tail circuits to the NAPs being congested is to get higher speed circuits. We would not be opposed to providing this info since it could be used to justify the need for higher speed access circuits to the NAPs.

Getting aggreement on how the information is presented and how often it is collected is going to be a big issue.

Date: Mon, 07 Apr 1997 13:36:43 -0400
From: Curtis Villamizar [curtis@ans.net] and Daniel McRobb [dwm@ans.net]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

Two reasons are proactive engineering measures and realistic self-assessment.

What type of investments would reap maximum value for the ISPs? for end users?

This changes depending on where the trouble spot of the moment is (moment being on the order of months or years). In the past (NSFNET ancient history, circa 1993-5) one problem was slow route convergence and the tendency of intermittent circuits to oscillate. LQM would have provided the means to take proactive measures when errors were very intermittent and avoid more significant outage. As circuit utilizations grow, monitoring link utilization and queueing loss can help isolate problem areas. This has always been an issue for tail circuits.

In order to fine tune traffic engineering, an AS to AS or prefix to prefix traffic matrix is needed. This was a capability lost with the NSS routers. Cflowd is an attempt to regain this capability. For IP over ATM networks, VC traffic statistics provide some capability. The capabilites are valuable but somewhat limited. For example, VC statistics cannot predict the impact of adding an AS peering (such as private peerings with AS exchanging high traffic volumes).

End to end performance measures can provide a realistic self-assessment. They can also be the basis for customer assurance that best effort service performance is of a high quality.

I don't propose to tell other providers how to run their business.

By end user, I think you mean institutions, such as universities, corporations, government agencies, and not all individuals. There is a need for end users at the extreme bottom of the food chain to have some measure of performance. A K-12 institution, or even an individual has a need to know whether the latencies they see should be regarded as "normal", meaning that they are due to bottlenecks near the server end or due to the limit of the link at the client end, or are due to their immediate service provider. This is all part of making an informed buying decision.

I'm going to answer the second part of your question, "What type of investments would reap maximum value for end users?", with "I don't know". There is a need, but I don't know what end user investment in organization, measurement tool, or cooperative measurement will reap the greatest benefit.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

Institutional end user needs and provider needs differ. What is shared in common is the need to be able to make measurements.

A major impediment to the ability to study traffic, both for provider engineering needs and for acedemic purposes is the inability to sample high speed data. Making the need for low level traffic sampling capabilities clearly known to equipment vendors is a an area where a stronger joint effort might yield improvements.

Some providers have expressed willingness to share traffic distribution information (traffic breakdown by service, flow duration, rate of new flow appearance, transfer size, etc) where releasing information is not sensitive. This has also proven helpful in aquiring an understanding of problems.

They should protect their own interests and look for ways to determine the quality of the service they are being provided. Clearly many have discovered that buying soley on cost may not be the best idea.

a. WAN measurement - I'll take this to mean measurements within the ISP cloud. LQM on high speed interfaces and on VCs would be nice. Making a variation of LQM an IP protocol rather than part of PPP would help for VCs and for NBMA and broadcast interfaces where LQM parameters could be associated with an ARP (or static-ARP) entry.

Providers need to cobble their own tools to gather queue loss statistics from existing MIBs and make sense of them.

b. measurement across ISP clouds - Providers can and some do make measurements across their own clouds.
End users might find it in their own interest to do the same. Cooperation among end users is needed for this.
Providers are in a better position to make measurement but it is in the providers best interest to provide statistics under non-disclosure and even assist direct customers in making their own measurements with the restriction of confidentiality.

c. traffic flow characterization - The raw data is not available from routers today. We need to convince vendors of the importance of these measurement interfaces.

d. network simulations - Network simulations involving realistic conditions push the scaling limits of currently available tools. Realistic conditions include high speed links, mix of short and long duration flows, mix of feeding bandwidths and delays, multiple congestion points, and very large numbers of traffic flows. Simulations today are pressed to explain what goes on today, let alone provide the means to experiment with improvements in queue management and packet scheduling. [how's that for a dose of e2e jargon:]

We store data in binary for speed of access and storage requirements. That seems to be a difference between what we do and efforts like netscarf. Might be a matter of scale since we used to get by with ASCII but were forced to abandon it.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

This is my first ISMA. To my knowledge much of the type of data the end users have wanted are the type that providers will only provide under non-disclosure.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

I personally like the idea of independent end user measurements. The nice thing about this is that it is unhamperred by proprietary concerns and there is greater assurance that measurement methodology is applied consistently.

As a professional I'm not as much a part of the end user community.

It never ceases to amaze me that some providers are hostile about the idea of independent measurements of their network performance.

You already know where to find cflowd (http://engr.ans.net/cflowd).

As far as recommending other publicly available tools, we're interested in tkinetd, though not using it yet. Netscarf will probably meet most ISPs needs though we home brewed our own over the course of years and like it. If you want to do simulations LBL's ns is great.

Date: Wed, 23 Apr 1997 01:48:00
From: Rick Wilder [wilder@mci.net]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis? What type of investments would reap maximum value for the ISPs? for end users?

for the ISPs: systems to support backbone traffic characterization including:

• traffic matrix of entry-exit points for the ISP's infrastructure.
• traffic trends in terms of quantity, time-of-day dynamics, growth rates for aggregate and individual protocol types.

For users

• systems to measure end-to-end performance, ie. available throughput, loss, and delay stats.

• Standarized data formats to promote sharability of measurements.
• development of generally useful systems for measurement/analysis

Develop specifications for internet service, including QoS requirements and share the with providers.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

• Lack of capability to measure effectively on high-rate (OC-3 and above) trunks and interconnect points.

• Lack of standard metrics and analysis techniques

• Lack of motivation to widely share data

• Scaling of measurement systems to higher capacity trunks

• Scaling of database/analysis systems to larger networks and more traffic.

The scaling factors listed above result in the need to reduce data.

Some analysis and research still requires great quantities of detailed data. Systems have to continuously support summary data from a large number of measurement points as well as temporary detailed measurements from a few measurement points

5. Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.

See material on OC3MON, Coral, etc. on www.nlanr.net and www.vbns.net.

Date: Tue, 15 Apr 1997 19:41:48 -0700 (PDT)
From: Bill Woodcock [woody@zocalo.net]

1. In your opinion, a. Why is it critical for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis?

I think that understanding traffic and utilization patterns is a necessary step in providing reliable service, and I tend to place reliability before performance or price, in developing our service offering.

What type of investments would reap maximum value for the ISPs? for end users?

That really depends what market segment the ISP is attempting to address. I think there's really very little place for statistical analysis, or research of any kind, in the $20-a-month dial-up market, for instance. But for NSPs and business-oriented leased-line providers, comprehensive logging facilities and packet-analysis capability would seem to me to be a minimum provision for dealing with such problems as come up.

b. The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?

No. There are other people with greater insight than I have already in a better position to guide progress.

c. What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?

Suggesting metrics and bringing problems to the attention of people designing tools, and hosting end-to-end measurement facilities.

2. What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?

It seems to me that we're having a bit of a problem of scale. :-) I think that perhaps many of the measurement problems that we're starting to tackle now might be more profitably addressed by biting off smaller chunks first, and designing tools which take the point-of-view of a single ASN, rather than an overview at shallower depth, then work on assimilating the data into a composite view.

3. What should the community (ISPs, research, users) be considering wrt... a. acquiring, storing and analyzing data?

Standard, interchangable, simple file formats, so that tools and datasets will be interoperable and exchangeable, and so that people will be able to more easily validate results using different processes or analytical methods.

b. level of detail of raw data and resulting analyses available to the general public? available to participating organizations?

Rather than handing raw ammo^H^H^H^Hdata to the press, I think perhaps quarterly or monthly position statements might be in order. These could contain or reference processed datasets, and would be less subject to misinterpretation by the press.

c. the appropriate organizational structures / strategies for collecting and analyzing infrastructure-wide traffic data?

This seems to me to be a simple matter of programming. :-)

4. Are there any specific measurement or analysis tools you feel are particularly useful? please describe.

I'd like to see work focused on AS-matrix flow analysis. Since all ISPs are ultimately interested in choosing the most efficient paths of ingress and egress for their data from a limited set, and there are a limited number of correspondent ASes, providing them with tools to collect and view this data seems to me to be a problem of addressable scope.

Last updated 30 April 1997
Comments should be addressed to tmonk@caida.org