Skip to Content
[CAIDA - Center for Applied Internet Data Analysis logo]
Center for Applied Internet Data Analysis
www.caida.org > projects : bwest : reports : toolEval.xml
Bandwidth Estimation Tool Evaluation
CAIDA is performing a series of experiments to evaluate and compare different bandwidth estimation tools. Evaluation methodology and results are reported below.
Outline
  1. Goals and objectives
  2. Testing Methodology
  3. Timeline
  4. Results to date
  5. Collaborators

Goals and objectives

GOAL: Compare and contrast different bandwidth estimation tools according to the same metrics.

Currently available bandwidth estimation tools measure different bandwidth metrics, use different methodologies and generate different amounts of overhead traffic. We wish to identify the strengths and weaknesses of each tool in order to provide a "Consumer's Report" style analysis that will allow researchers and application developers to choose an appropriate tool to meet their needs.

OBJECTIVE: Characterize tools and prioritize order of evaluations. Note: tools in parentheses haven't yet been installed.
  1. Per-hop (link) capacity tools: pathchar, pipechar, pchar, clink

  2. End-to-end capacity tools: Sprobe, pathrate, nettimer, (bprobe)

  3. End-to-end available bandwidth tool: (pathload)

  4. BTC or TCP throughput tools: (bbcp, cap)

  5. Parallel TCP connections throughput tools: Iperf, (netperf)

OBJECTIVE: Baseline tool measurements against simulated traffic.
  1. Measure on known capacity links

  2. Measure while increasing link utilisation in 5% steps (via Smartbits traffic generator)

  3. Measure while injecting different types of cross-traffic:

    • constant bitrate fixed size UDP packets (1500 B)
    • random mix of 40/576/1500 B packets
    • single large TCP stream
    • 5/10/20 random-size file transfer streams
OBJECTIVE: Characterize tool measurements against real traffic.
  1. Measure/estimate UCSD traffic for 24 hr weekday on known path

  2. Measure/estimate Abilene traffic for 24 hr weekday on known path

  3. Read router SNMP counters in parallel with above .


Testing Methodology

Testlab configuration (simulated traffic)

  • Smartbits6000 Traffic Generator configured by console PC connected to its backend control port
  • Host wednesday: NeTraMet flow meter/passive monitor atttached to 100M load path via 100Mb copper tap and to GigEther load path via fiber repeater
  • Cisco Catalyst 2900 switch
  • Extreme Networks Summit 5i router
  • Juniper M20 router
  • Foundry Big Iron router
  • Host bw-est-1: CAIDA 100M bwest test box A (Sender)
  • Host bw-est-2: CAIDA 100M bwest test box B (Receiver)
  • Host fester: CAIDA GigEther bwest test box A (sender)
  • Host gomez: CAIDA GigEther bwest test box B (Receiver)
  • n pairs of network interfaces managed by Smartbits:
    1. 10/100/1000 Ethernet
    2. GigEther (fiber)
    3. OC48 (1 pair)

Testlab configuration (real traffic, with known switches and routers)

  • NeTraMet flow meter/passive monitor
  • mesh of bwest and re-purposed skitter boxes:
    1. CAIDA bwest test box A (???)
    2. CAIDA bwest test box B (???)
    3. chenin (Boulder, CO)
    4. champagne (Illinois)
    5. zinfandel (???)
    6. merlot (???)

Timeline

TaskDatePeople
Configure Reference Lab with 10/100M NICs at SDSC12Mar02Kevin, marg, vineet
Run tools against simulated traffic in closed labApr-Jun02vineet
Analyze and graph resultsApr-Jun02vineet, marina
Configure end-host on UCSD path(s)Jul02Kevin, marg, vineet
Run tools from machine in CalNGI lab to UCSD end host. Track level of real cross-trafficJul-Aug02vineet,ravi
Analyze and graph resultsAug02vineet,marina
Configure end-host on Abilene path(s) Aug02ravi
Run tools on Abilene path using real cross-traffic Aug-Sep02vineet, ravi
Analyze and graph resultsSep02vineet, marina
Draft of results paperAug1502vineet, ravi, marg
Draft of sysadmin/netops paperOct02vineet,marg

Results to date

Vineet's Initial measurement results
Paper: Comparison of Public End-to-End Bandwidth Estimation Tools on High-Speed Links


Collaborators

Kevin Walsh and CalNGI Reference Test Lab

Les Cottrell's PingER data

Martin Swany's NWS bandwidth forecaster

  Last Modified: Wed Mar-27-2019 22:23:19 PDT
  Page URL: http://www.caida.org/projects/bwest/reports/toolEval.xml