• HOME
  • About
  • Conference Calendar
  • Jobs
  • Legal Agreements
  • Staff
  • Joining CAIDA
  • How-To
  • Program Plan
• RESEARCH
  • Routing
  • Topology
  • DNS
  • Security
  • Traffic Analysis
  • Policy
  • Visualization
• DATA
  • Overview
  • Data Sharing
  • How-to
  • Papers using CAIDA Data
• TOOLS
  • Measurement
  • Taxonomy
  • Utilities
  • Visualization
• PUBLICATIONS
  • Papers
  • Presentations
  • Animations
  • Visualizations
  • Bibliography
  • Posters
• WORKSHOPS
  • ISMA
  • WIDE
  • External
  • Archive
• PROJECTS
  • Macroscopic Topology
  • PREDICT
  • Day in the Life
  • IMDC
  • Ark
  • Network Telescope
  • Coralreef
  • IIC Wiki
  • COMMONS
  • IPNC
• FUNDING
  • Cybersecurity
  • Cisco
  • CONMI
  • DNS-ITR
  • NeTS-FIND
  • PREDICT
  • WIDE
  • Program Plan

*** DRAFT ***

Preliminary Measurement Spec
for Internet Routers

This strawman measurement specification is for review and comment purposes only.

The Cooperative Association for Internet Data Analysis (CAIDA) is editing this specification based on inputs from: Bell-Labs, Qwest, Verio, Worldcom, Zocalo...

Comments on this draft measurement specification should be sent to Tracie Monk, tmonk @ caida.org

The most updated copy of this document is posted at http://www.caida.org/tools/measurement/measurementspec/

A mailing list has been set up for the discussion of this document, at meas-spec@caida.org.
To subscribe or unsubscribe, send mail to:

with one of the following in the in the body of the message:

subscribe
subscribe username@host.domain

unsubscribe
unsubscribe username@host.domain

1.0 Introduction

Collection and analysis of basic traffic statistics is fundamental to providers' ability to design and and operate their networks. In addition to link utilization statistics (e.g., data supplied by MIB- II and other SNMP MIBs.[1]), both long-term aggregated statistics and short-term per flow statistics provide necessary insights relating to

• network provisioning
• peering arrangements
• per-customer accounting, SLA verification
• per-peer accounting (traffic balance of trade)
• performance management
• tracking topology and routing changes
• tracing back DOS attacks
• ATM/cell/circuit level errors
• other trouble shooting
• connectivity complexity/vulnerability
• TCP flow dynamics
• routing table/address space efficiency

Backbone engineering and planning are among the most pressing needs for reliable forms of traffic data and analyses. Key elements of these analyses are aggregate traffic data at the IP layer, including port and protocol statistics (packets and bytes per port and per protocol) and traffic matrix statistics (how many packets and bytes were sent from network A to network B). At the 1997 Internet Statistics and Metrics Analysis (ISMA) workshop, however, participants criticized both the vendor community (for not addressing these needs) and the research community for not assisting to articulate requirements associated with traffic collection and analysis, saying...

One participant from ANS described what forms of measurement data that he and other backbone engineers need most, suggesting that critical data not readily available include:

A year earlier, at the 1996 ISMA, Dennis Ferguson (Juniper) described one of ISPs' most noteable Achilles heels, saying that in 10 years there has not yet developed a basic Internet traffic engineering methodology. If even half the attention to `rocket science traffic modeling' were devoted to how to estimate a reasonable ingress-egress traffic matrix from sampled measurements, he explained, network engineers, particularly of large clouds, would find their job substantially easier. And yet, not even matrix estimates are available with most router equipment right now; link loads are available, but they just do not hold enough information to estimate a reasonable matrix. Ferguson suggested that deriving better estimates from samples and averages, the way civil engineers have derived approximations for highway architectures, would be a good first step. Simple rough traffic matrices would, at a minimum, improve a network engineer's ability to balance load and engineer topology.

This document focuses on the forms of traffic data needed by Internet Service Providers (ISP) for planning and traffic engineering purposes and identifies alternative forms of traffic data required for research. It is intended as a top-level specification for vendors to use in developing traffic collection features of Internet routers. It does not provide detailed information about the implementation of individual vendor solutions. This specification reflects the comments and suggestions by network engineers and researchers. It also builds on the excellent foundation provided by Cisco through NetFlow™ and its ability to export the flow data for external analysis.

2.0 Architecture Considerations

2.1 Flow Definition

A common definition of a "traffic flow" is important to data aggregation and to comparisons of traffic statistics across various router vendor traffic collection schemes. Comparison of these router-derived data with data collected from independent measurement mechanisms such as the Coral monitors (see http://www.caida.org/tools/measurement/coralreef/) or NetraMet (see http://www.auckland.ac.nz/net/Internet/rtfm/) is also desirable. The sections below describe key parameters influencing vendor adoption of a definition of "traffic flows" for measurement purposes. [Note that some routers use "flows" for packet forwarding decisions; a definition of flows for measurement purposes will have to take alternative requirements into account.]

Directionality: For the purpose of traffic measurements, flows are either unidirectional (A to B) or bidirectional (A to B and B to A with no distinction in directionality). A unidirectional definition is generally used as the default: traffic from A to B and from B to A (including acknowledgements) are considered separate traffic flows for aggregation and analysis purposes. The availability of bidirectional data is also important, however. Bidirectional data provides insights into the behavior of individual protocols, including problems that may manifest themselves in core backbones, but which are harder to identify at endpoints.

Endpoint Aggregations and Granularity: In his thesis on using flows for aggregation and measurement of traffic data, Siegfried Loffler explains that

For ISPs, the primary endpoint identifier of a flow is the end-point ASes. Making the endpoints user-definable, however, significantly expands the breadth of available analysis options.

The available information on which to build a flow identifier tuplet will vary, but at a minimum, the following would typically be needed for IPv4:

• source host IP address
• destination host IP address
• transport protocol
• source port (where applicable)
• destination port (where applicable)
• start time of flow
• end time of flow

These parameters are enough to uniquely identify a flow at the transport layer. Note that one can not universally identify flows at the application level from this information; while you can assume particular applications for well-known ports, it is always a little inexact, particularly for UDP and any TCP channel that's multiplexed at the application layer. Multicast is even more complicate (see section on multicast at end of this document.)

Timeout: For network measurement and analysis, the focus of the flow definition needs to be on defining a unit of time permitting adequate traffic aggregation into `flows'. A flow is considered active so long as incoming packets are not separated in time for more than the length of the interval between two timeout checkpoints.

Opinions about the acceptable length of the flow timeout interval vary from seconds to minutes depending on analysis requirements.[2] Research by Claffy, Braun, and Polyzos on the NSFnet backbone [1995] suggested that at that time, typical flows between two hosts on the Internet rarely lasted longer than ten seconds. Their research suggested that with timeout values of 64 seconds or less, 90% of the flows showed less than 50 packets, 5.5 kilobytes and 100 seconds of duration. In the absence of a less arbitrary definition, the 64 seconds timeout value has been implemented as the default in the Coral monitors (OC3mon and OC12mon). Alternatively, the NetraMet monitor uses a default timeout value of 600 seconds and Cisco's NetFlow™ uses a default timeout of 30 minutes (among other mechanisms for timing flows out, such as needing to reclaim memory). In each case, the timeout value is user configurable, an important feature required to tune the trade off between traffic volume and memory, e.g., a measurement point experiencing a high number of flows may require a shorter timeout value than a minimally loaded link.

A mechanism for expiring long-lived flows should be user configurable since it can affect reported results by producing artificial traffic spikes (you have to force timeouts for 'continuous flows' eventually, which means choosing some arbitrary cutoff point.) A 5-minute granularity is suggested for artificially expiring (timing out) long-running flows. It is also important for reporting purposes that flows be terminated before any counter rollover occurs. Whether a vendor chooses to implement counters that use lots of memory to hold flows for a long time or use counters that necessitate frequent export of data is up to the vendor. But an analysis application should not have to guess at whether or not a counter has rolled over.

Flow Definition: The following definition of "flow" is recommended for use in the development of future measurement specifications for Internet routers

The figure below illustrates @@@@@ add flow schmeatic ...

2.2 Methods

Processor: A fundamental trade-off exists between the flexibility (and cost saving) of having measurements provided on the router versus potential performance costs associated with enabling the statistics collection features of Internet routing/switching hardware. Processing of traffic flows can be relatively memory intensive for the main processor. Under heavy traffic loads, a router using features associated with the forwarding path (as opposed to a card that may sniff router backplane rails) may experience performance degradations.

Ideally, the measurement functionality of routers should be implemented in a manner that:

• enables customers to be able to collect the data without undue impact on forwarding performance of the router, and
• provides a concise, clear means of acquiring significant quantities of accurate data

Of primary importance in the near term is providing customers with routers capable of exporting critical aggregated data, such as AS matrices, that are needed for capacity planning.

Sampling: While some measurement devices, e.g., the Coral monitors, may be capable of continuous monitoring of traffic flows, applications such as capacity planning do not require this level of detail. During the NSFNET era, ANS Communications Inc. conducted tests on sampling at granularities of 1 in 50 packets, 1 in 100 packets, and 1 in 1000 packets. For the purposes of capacity planning, they found that sampling rates of 1 in N packets is acceptable for moderate values of N. Sampling rates of as high as 1 in 1000 packets require confidence intervals that are too high to trust for capacity planning. Sampling based on time intervals is also discouraged since the results do not safely reflect the burstiness characteristics of traffic.

• a random choice of 1 of N where a random member out of the next N packets is chosen -- thereby reducing biases introduced into the data by sampling
• selection of consecutive frames/packets for interarrival time studies -- for analysis of packet arrival times , data reflecting bursts of back-to-back packets is needed
• selection of every frame/packet from a single flow (or possibly just headers) -- this approach is particularly relevant for TCP dynamics or other protocol analyses

Additional research is needed regarding acceptable sampling rates and confidence intervals for specific application requirements. In the interim however, this parameter should be user-definable with increasingly fine granularity as customers move from requirements associated with capacity and architecture planning to those associated with accounting and billing.

2.3 Address Space Analysis

A router measurement implementation should support the granularity of classless networks using a CIDR-aware IP-to-AS-path mapping that is current from the data collection point perspective. This includes summarizing the flow data per AS, a more convenient macroscopic level. Indeed, AS-to-AS matrices are critical to rational capacity and peering engineering.

2.4 Flow Export Formats

Indexing - Ideally, a router's flow export format should employ a system whereby each flow packet is indexed describing its contents, and routers are configured with knobs permitting users to specify which data fields they would like exported. This way a single code path could cover any of the possible aggregations with minimal switch statements (e.g., new structures and code to handle those structures).

The fields would be written to a raw file in the order that they appear in the index bitmask, and then those saved flow records could be specified using a bitmask as an index, e.g., if a bit is on in the index, a particular piece of data is present; if it is off, it's not present. The entry in the flow export packet would only write to those fields identified in the index. (The indicator bits in the index would determine the write order.) Using indexing should be comparable (in terms of load upon the router) to export solutions such as NetFlow™, but indexing should simplify flow collection and provide flexibility in terms of customer-specified tuplet combinations and alternative aggregation schemes. Indexing also makes for easier integration of export flow data with other things which might be capable of generating flow data, e.g., RMON probes, low-end routers that don't run BGP, Coral monitors, etc. Note one could also do nexthop matrices, interface matrices (very RMON-like), peer_nexthop matrices, etc. with this scheme. Vendors would also save bandwidth since only requested data would be exported.

An illustrative bitmask index includes:

Routers should export available data in a manner facilitating users' ability to identify the export data with multiple bits set in the export index directly associating the output, i.e., a flow whose identifier is the tuplet of each of the things whose bit is set in the index. Similarly, routers should export AS path information within flow records as unique integer identifiers, and have the router separately export the actual AS path/id mapping as it learns about new AS paths. Since the router already has to have a table of AS paths this should not be too complicated, and would keep the flow record size manageable.

Note that all export data should include the IP address of source from which the data is collected, identify the link (or LAN) from which the data was taken, and include the timestamp of the flow.

Tuplet Specification - User configurable tuplet specification should be enabled as a measurement aggregation feature. Priority parameters include:

• IP header TOS field
• input interface index
• source network mask (CIDR)
• destination network mask (CIDR)
• source AS path (as opposed to origin AS or peer AS)
• destination AS path (as opposed to origin AS or peer AS)
• bitwise OR of all TCP flags seen in a flow

Transport Mechanism - Unacknowledged UDP transmissions of flow export data can not guarantee that all data being broadcast is received by the collection/management station. In addition, accurate analysis is complicated by the possibility of dropped packets (e.g., a packet indicating a changed AS path) or when data is put in the wrong bucket. In general, alternative export transportation solutions utilizing TCP may be more reliable than UDP transport.

2.5 Timestamp

Timestamps should be in milliseconds or better. Epoch should be 00:00:00.00 GMT, Jan. 1, 1970.

2.6 Other

Suggestions as to other aspects of traffic that merit monitoring and analysis include:

3.0 Aggregation Statistics

3.1 Basic Counters

Basic measurement counters used in traffic collection and analysis consist of bytes, packets, and flows. (See RFC 2233 for a definition of normal counters, RMON-I counters, and error stats.)

In terms of this measurement spec, it is important that routers be capable of exporting the following additional data associated with specific ASes or hosts or as a full matrix:

• flows received - total flows received by router
• indicator of data loss, e.g., indicator of the number and size of flows missed from router
• protocol traffic - packet and byte counters for each IP protocol (TCP, UDP, etc.)
• port traffic - keyed by the tuplet protocol:srcport:dstport
• net traffic - packets and bytes sent from one network to another
• AS traffic - packets and bytes sent from one AS to another
• router interface

[cflowd-specific: Note a 'missed flows' counter may have some inaccuracies due to flow export packets arriving out of order to the collector. This is an issue with UDP transport and with the current implementation of cflowd when multiple equal-cost paths exist between the router and the collector.]

3.2 Aggregation Tables and Matrices

The counters described above are required for the development of aggregation tables and matrices. According to inputs from ISPs, the most critical aggregation scheme for network planning and engineering are:

• protocol tables
• port tables (source and destination)
• matrix keyed by protocol:srcport:dstport
• peer AS:AS matrices
• Net:Net matrices

Additional aggregation schemes useful in network planning and as inputs for software and hardware vendors include:

• origin AS:AS matrices
• host matrices (inc. port & protocol)
• next hop tables (inc. input/output interface numbers and IP addresses)
• heavy hitter tables
• MAC address tables (useful for multicast, tho latter also needs oif list info)

Lower level features, similar to those implemented in NetFlow), may also be used in the flow collector or management modules, however, they necessitate collection and retention of large quantities of data at varying levels of granularity. For entities requiring detailed data analysis, drill-down features facilitate review of secondary data associated with a primary aggregation scheme, e.g., elements of a detailed AS:AS matrix might be expanded to reveal additional data such as IP numbers or next hop.

Illustrative table and matrix outputs using cflowd and NetFlow™, Coral/OC3mon, and NetraMet are provided in Appendix A. @@ examples

3.3 Special statistics needed for multicast

Multicast traffic presents a unique problem, as traffic from a single flow will typically be sent out multiple interfaces during the course of normal operation. Keeping a record of the output interfaces and tracking changes in this list is the primary additional requirement for measuring multicast traffic.

The output interface lists for a specific multicast group will typically be constant over a period of several minutes. Consequently, it will likely be sufficient to report the union of all of the output interfaces that were active during the flow's lifetime as the output interface list when the flow is expired.

An index of the output interface lists analogous to the index of AS paths discribed elswhere in this document should be maintained and periodically announced to the collector. Ideally, this index should include the next hop(s) on each interface that are requesting the multicast traffic so that the distribution tree of a multicast flow can be reconstructed.

4.0 Service Level Agreements

Research is still required on what passive indicators, likely in conjunction with other active statistics and tools for correlating between them, might support maintenance of Service Level Agreements. For example, a passive listening device might record timestamp deltas measured in sniffed packets, which one could then compare to data from active tools or SNMP statistics.

5.0 Futures

How will MPLS change these recommendations?

How will IPSEC change these recommendations?

How will all-optical networks change these recommendations?

6.0 Business and Legal Considerations

The purpose of this document is to provide a technical foundation for measurement functionality in Internet routers. However, given the growing concern regarding privacy and ISP liabilities relating to monitoring, a relevant quote from a Bellcore lawyer to issues discussed at ISMA'96 is included below.

Definitions

@@@ ADD

Footnotes:

[1] SNMP monitoring tools include the MultiRouter Traffic Grapher (MRTG) by Tobias Oetiker, ETH Zentrum (see: http://ee-staff.ethz.ch/~oetiker/webtools/mrtg/mrtg.html); RTRmon by Vixie and Associates (see: http://www.vix.com/rtrmon/); and Vulture by Vixie and Associates (see: http://www.vix.com/vulture/).

References

Acharya M. and B. Bhalla, A flow model for computer network traffic using real-time measurements, in Second International Conference on Telecommunications Systems, Modeling and Analysis, March 24-27 1994.

Caceres, R., P. Danzig, S. Jamin, and D. Mitzel; Characteristics of wide-area TCP/IP conversations, in Proceedings of ACM SIGCOMM '91, September 1991, pp. 101--112.

Claffy, K., George Polyzos and Hans-Werner Braun, A Parametrizable methodology for Internet traffic flow profiling, Mar 1995, IEEE JSAC Special Issue on the Global Internet.

Claffy, K., Greg Miller and Kevin Thompson, The Nature of the Beast: Recent Traffic Measurements from an Internet Backbone, INET'98; July 1998. http://www.caida.org/publications/papers/1998/Inet98/

Danzig, P., S. Jamin, R. Caceres, D. Mitzel, and D. Estrin, An Empirical Workload Model for Driving Wide-Area TCP/IP Network Simulations, Journal of Internetworking: Research and Experience, vol. 3, n. 1, pp. 1-26, March 1992.

Estrin, D. and D. Mitzel, An assessment of state and lookup overhead in routers, in Proceedings of IEEE Infocom 92, May 1992, pp. 2332--42.

ISMA 1997: Workshop Report at http://www.caida.org/workshops/isma/9705/isma97_report.html

ISMA 1996: Report of the NSF-sponsored workshop on Internet statistics measurement and analysis at http://www.caida.org/workshops/isma/9602/report/

Loffler, Siegfried, Institute of Communication Networks and Computer Engineering (IND) of the University of Stuttgart, Using Flows for Analysis and Measurement of Internet Traffic; Diploma Thesis, August 4, 1997; http://www.mathematik.uni-stuttgart.de/~floeff/diplom/report/diplom.html

RFCs:

2233 K. McCloghrie and F. Kastenholz, "The Interfaces Group MIB using SMIv2", November 1997.
http://www.cis.ohio-state.edu/htbin/rfc/rfc2233.html

2123 N. Brownlee, "Traffic Flow Measurement: Experiences with NeTraMet", 03/28/1997.
http://www.cis.ohio-state.edu/htbin/rfc/rfc2123.html

2021 S. Waldbusser, "Remote Network Monitoring Management Information Base Version 2 using SMIv2", 01/16/1997.
http://www.cis.ohio-state.edu/htbin/rfc/rfc2021.html

2074 A. Bierman, R. Iddon, "Remote Network Monitoring MIB Protocol Identifiers", 01/16/1997.
http://www.cis.ohio-state.edu/htbin/rfc/rfc2074.html

2064 N. Brownlee, "Traffic Flow Measurement: Meter MIB", 01/03/1997.
http://www.cis.ohio-state.edu/htbin/rfc/rfc2064.html

2063 N. Brownlee, C. Mills, G. Ruth, "Traffic Flow Measurement: Architecture", 01/03/1997.
http://www.cis.ohio-state.edu/htbin/rfc/rfc2063.html

1953 W. Edwards, F. Liaw, T. Lyon, G. Minshall, "Ipsilon Flow Management Protocol Specification for IPv4 Version 1.0", 05/23/1996.
http://www.cis.ohio-state.edu/htbin/rfc/rfc1953.html

1700 J. Reynolds, J. Postel, "ASSIGNED NUMBERS", 10/20/1994. (Obsoletes RFC1340) (STD 2)
http://www.cis.ohio-state.edu/htbin/rfc/rfc1700.html

1363 C. Partridge, "A Proposed Flow Specification", 09/10/1992.
http://www.cis.ohio-state.edu/htbin/rfc/rfc1363.html

1213 K. McCloghrie, M. Rose, "Management Information Base for Network Management of TCP/IP-based internets: MIB-II", March 1991.
http://www.cis.ohio-state.edu/htbin/rfc/rfc1213.html