scamper — interact with scamper processes and data

Introduction¶

scamper is a tool that actively probes the Internet in order to analyze Internet topology and performance. The scamper tool provides rich functionality, this scamper module provides convenient classes and methods for interacting with scamper processes and data. The module has two related halves – those for interacting with running scamper processes (through ScamperCtrl and related classes) and those for reading and writing data previously collected with scamper (ScamperFile). These classes are supported by other classes that store measurement results. The types of measurements supported by this module include ping, traceroute, alias resolution, DNS queries, HTTP, UDP probes, and packet capture.

Interacting with Scamper Processes¶

Simple Parallel Measurement¶

The following example implements the well-known single-radius measurement technique, which conducts delay measurements to an IP address from a distributed set of vantage points, and reports the shortest of all the observed delays with the name of the vantage point that observed the delay. The ScamperCtrl object is instantiated with a single parameter that identifies a directory that contains a set of Unix domain sockets, each of which represents a vantage point running scamper; the ScamperCtrl object makes these vantage points available via instances(), which the caller uses to conduct a ping measurement via each of them. These ping measurements operate in parallel – the ping measurements on each of the nodes operate asynchronously. We then collect the results of the measurements, noting the minimum observed delay, and the vantage point where it came from. We pass a 10-second timeout to responses() so that if a vantage point experiences an outage after we send the measurements, it would not hold up the whole experiment. Finally, we print the result of the measurement.

import sys
from datetime import timedelta
from scamper import ScamperCtrl

if len(sys.argv) != 3:
  print("usage: single-radius.py $dir $ip")
  sys.exit(-1)

ctrl = ScamperCtrl(remote_dir=sys.argv[1])
for i in ctrl.instances():
  ctrl.do_ping(sys.argv[2], inst=i)

min_rtt = None
min_vp = None
for o in ctrl.responses(timeout=timedelta(seconds=10)):
  if o.min_rtt is not None and (min_rtt is None or min_rtt > o.min_rtt):
    min_rtt = o.min_rtt
    min_vp = o.inst

if min_rtt is not None:
  print(f"{min_vp.name} {(min_rtt.total_seconds()*1000):.1f} ms")
else:
  print(f"no responses for {sys.argv[2]}")

Reactive Measurement¶

The next example shows a more complex measurement. Let’s say we want to know the RTTs to the authoritative name servers for a zone from a single vantage point. This implementation takes two parameters – a single vantage point, and a zone name to study. We open an interface to that VP, and then issue a DNS query for the authoritative name servers for the zone from that VP. There are a couple of interesting things to note. First, we do not pass a handle representing the VP instance to the do_dns() method, as the ScamperCtrl interface only has a single instance associated with it – it is smart enough to use that single instance for the measurement. Second, we pass sync = True to make the measurement synchronous – the method does not return until it has the results of that single measurement. This is shorter (in terms of lines of code) and more readable than issuing an asynchronous measurement and then collecting the single result. Then, we issue asynchronous queries for the IPv4 and IPv6 addresses for the name servers returned and send ping measurements for each of the addresses. Finally, we print the names of the nameservers, their IP addresses, and the minimum RTT observed to each.

import sys
from datetime import timedelta
from scamper import ScamperCtrl

if len(sys.argv) != 3:
  print("usage: authns-delay.py $vp $zone")
  sys.exit(-1)

ctrl = ScamperCtrl(remote=sys.argv[1])

# get the list of NS for the zone
o = ctrl.do_dns(sys.argv[2], qtype='NS', wait_timeout=1, sync=True)

# issue queries for the IP addresses of the authoritative servers
ns = {}
for rr in o.ans():
  if rr.ns is not None and rr.ns not in ns:
    ns[rr.ns] = 1
    ctrl.do_dns(rr.ns, qtype='A', wait_timeout=1)
    ctrl.do_dns(rr.ns, qtype='AAAA', wait_timeout=1)

# collect the unique addresses out of the address lookups
addr = {}
for o in ctrl.responses(timeout=timedelta(seconds=3)):
  for a in o.ans_addrs():
    addr[a] = o.qname

# collect RTTs for the unique IP addresses
for a in addr:
  ctrl.do_ping(a)
for o in ctrl.responses(timeout=timedelta(seconds=10)):
  print(f"{addr[o.dst]} {o.dst} " +
        (f"{(o.min_rtt.total_seconds() * 1000):.1f}"
         if o.min_rtt is not None else "???"))

Dynamic Event-driven Measurement¶

The module also supports a more dynamic approach to interacting with many vantage points that each signal their desire for more work. The following code maintains a list of addresses per vantage point to ping. The vps dictionary maintains a mapping of ScamperInst to that list of addresses. When each instance signals that it is ready for more work, the morecb is called, with the overall ScamperCtrl, the specific ScamperInst that wants another measurement, and the param specified to the ScamperCtrl constructor.

import sys
from scamper import ScamperCtrl
from datetime import timedelta

@staticmethod
def _feedme(ctrl, inst, vps):
  if len(vps[inst]) == 0:
    inst.done()
  else:
    ctrl.do_ping(vps[inst].pop(0), inst=inst)

@staticmethod
def _main(vps):
  vps = {}
  ctrl = ScamperCtrl(morecb=_feedme, param=vps)
  for vp in vps:
    inst = ctrl.add_remote(vp)
    vps[inst] = ["192.0.2.1", "192.0.2.2", "192.0.2.3"]

  # issue measurements as each VP asks for a new measurement
  while not ctrl.is_done():
    o = None
    try:
      o = ctrl.poll(timeout=timedelta(seconds=10))
    except Exception as e:
      print(f"got exception {e}")
      continue

    # if ctrl.poll() returns None, either all measurements are
    # complete, or we timed out.  say what happened.
    if o is None:
      if ctrl.is_done():
        print("done")
      else:
        print("timed out")
      break

    print(f"{o.inst.name} {o.dst} {o.min_rtt}")

  return 0

if __name__ == "__main__":
  sys.exit(_main(sys.argv[1:]))

The module supports many different types of measurements:

Traceroute via do_trace()
Ping via do_ping()
Alias resolution via do_ally(), do_mercator(), do_midarest(), do_midardisc(), do_prefixscan(), and do_radargun()
DNS via do_dns()
Packet capture via do_sniff()
MDA traceroute via do_tracelb()
HTTP via do_http()
UDP probes via do_udpprobe()

Reading and Writing Files¶

To read results stored in a native scamper warts(5) file, instantiate a ScamperFile object, passing the name of the file to open as the first parameter. Read each object out of the file using the read() method, or by using the built in Iterator, demonstrated below. If the file contains objects of different types, but you are only interested in a subset of the object types, you can signal the types with the filter_types() method, or by signalling that with the constructor. Finally, you can close the file when you are finished using the close() method.

The following code illustrates the overall approach:

from scamper import ScamperFile, ScamperTrace

addrs = {}
file = ScamperFile("foo.warts.gz")
for o in file:
  if isinstance(o, ScamperTrace):
    for hop in o.hops():
      if hop.addr is not None:
        addrs[hop.addr] = 1
file.close()

for o in sorted(list(addrs)):
  print(o)

This code reads ScamperTrace objects out of foo.warts.gz, storing addresses observed in each traceroute in a dictionary to identify the unique addresses. Finally, it prints the addresses in sorted order.

To write measurements to a file, instantiate a ScamperFile object, passing ‘w’ as the mode parameter. By default, ScamperFile will write warts(5) output, but this can be changed either by specifying the kind parameter, or by using an appropriate suffix in the filename. ScamperFile can write compressed warts(5) output using “warts.gz”, “warts.bz2”, or “warts.xz”, write json output with “json”, or simple text output with “text”. Write each object out by passing the object to the write() method.

If you wish to save measurement results collected via a ScamperCtrl, then open a file, and pass it to the ScamperCtrl constructor.

from scamper import ScamperCtrl, ScamperFile

outfile = ScamperFile("foo.warts.gz", 'w')
ctrl = ScamperCtrl(remote_dir="/tmp/dir", outfile = outfile)

This way, ScamperCtrl will record all measurement objects in the file, so that you do not have to write() them yourself.

API Reference¶

Classes for Managing Scamper¶

`ScamperCtrl`¶

class scamper.ScamperCtrl(meta=False, morecb=None, eofcb=None, param=None, unix=None, remote=None, remote_dir=None, outfile=None)¶

ScamperCtrl objects provide an event-driven interface to one or more scamper processes, each of which is represented by a single ScamperInst object. The general workflow is to instantiate a ScamperCtrl object connected that manages one or more ScamperInst objects, issue measurements via the various do methods, read and save results, and either issue new measurements or finish.

The ScamperCtrl constructor takes the following named parameters, all of which are optional. The meta parameter signals whether the caller wants meta objects (ScamperList, ScamperCycle) or not; the default is False. The morecb parameter is a callback function that ScamperCtrl will call when an scamper instance signals that it wants more work. The callback takes three parameters: a ScamperCtrl, a ScamperInst, and a user-supplied parameter that is passed to the callback to use. The eofcb parameter is a callback that ScamperCtrl will call when an instance signals that it has finished. The callback takes the same three parameters as morecb. The param parameter is the user-supplied parameter that will be passed to morecb and eofcb. The outfile parameter identifies a ScamperFile opened for writing that should record all measurement results. This can save the caller from writing code to store individual measurement results as these are passed to the caller. The unix parameter identifies the path in the file system to a unix domain socket representing a local scamper instance, which will then become accessible via a ScamperInst. The remote parameter identifies the path in the file system to a unix domain socket representing a remote scamper instance, which will then become accessible via a ScamperInst. The remote_dir parameter identifies the path to a directory in the file system containing multiple unix domain sockets, each of which represent a remote scamper instance that will become accessible via ScamperInst objects.

These first set of methods and attributes allow a program to manage and use a set of ScamperInst.

add_unix(path)¶

Add a single ScamperInst which represents a local scamper process available via a Unix domain socket at the specified location in the file system identified with the path parameter. The method returns the ScamperInst object to the caller. You do not have to store the object, as the ScamperCtrl object also retains a reference to the ScamperInst.

You can also connect a local scamper process when constructing the ScamperCtrl object by passing the path to the unix parameter in the constructor. This will save you writing code.

Raises a RuntimeError if the path is invalid, or the caller does not have sufficient privileges in the file system to open the socket.

add_inet(port, addr=None)¶

Add a single ScamperInst which represents a scamper process available on a regular socket. The port parameter is mandatory. The addr parameter is optional. If not supplied, the port is assumed to refer to a local scamper instance which can be found on the loopback IP address. The method returns the ScamperInst object to the caller. You do not have to store the object, as the ScamperCtrl object also retains a reference to the ScamperInst.

Raises a RuntimeError if the port is invalid, or the caller cannot connect to the port.

add_remote(path)¶

Add a single ScamperInst which represents a scamper process on a remote system available via a Unix domain socket at the specified location in the file system identified with the path parameter. The method returns the ScamperInst object to the caller. You do not have to store the object, as the ScamperCtrl object also retains a reference to the ScamperInst.

You can also connect a remote scamper process when constructing the ScamperCtrl object by passing the path to the remote parameter in the constructor. This will save you writing code.

Raises a RuntimeError if the path is invalid, or the caller does not have sufficient privileges in the file system to open the socket.

add_remote_dir(path)¶

Add a ScamperInst for each remote system available via a Unix domain socket in the specified directory in the file system identified with the path parameter.

You can also connect these systems when constructing the ScamperCtrl object by passing the path to the remote_dir parameter in the constructor. This will save you writing code.

Raises a RuntimeError if the path is invalid, or the caller does not have sufficient privileges in the file system to open a socket.

instances()¶: Return a list of ScamperInst managed by the ScamperCtrl. This provides a convenient interface to issue commands to all available vantage points.

instc¶: The total number of ScamperInst managed by this ScamperCtrl.

The following methods provide interfaces for obtaining measurement results, as well as handling exceptions.

poll(timeout=None, until=None)¶

Wait for a measurement result to become available from one of the ScamperInst under management. The method will block until a result is available, or an exception occurs. The timeout parameter limits the time the method will block before returning, and is a timedelta object. The until parameter specifies when the method should return, if there is no result available by then, and is a datetime object.

If an exception has occurred asynchronously, perhaps because a ScamperInst rejected a command, or a user-provided callback function raised an exception, this method will raise it.

responses(timeout=None, until=None)¶

Return all measurement results for all issued measurements, blocking until there are no issued measurements outstanding. The caller can limit the time this method will block by using the timeout and until parameters. The timeout parameter is a timedelta that specifies the length of time the responses generator can block before returning. The until parameter is a datetime that specifies when the method should return.

This method will not raise any exceptions that get queued asynchronously by ScamperCtrl. The caller should call the exceptions() method to obtain exceptions after responses() returns.

exceptions()¶: This method returns queued exceptions. You should call this method after calling responses().

done()¶: Signal that there is no further measurements to be issued on any of the connected ScamperInst.

is_done()¶: Returns True when all issued measurements have returned and all ScamperInst have signalled end-of-file.

taskc¶: The total number of tasks outstanding.

The following methods provide interfaces for issuing measurements. Each of the ScamperCtrl do_ methods have three parameters in common, which we document here for brevity. The first, sync, determines whether the measurement operates synchronously or not. By default, a measurement operates asynchronously (sync = False), and the do_ methods return a ScamperTask object that represents the measurement. When operating synchronously (sync = True), the do_ methods each return the result of the measurement when it becomes available. The second parameter in common is the inst parameter. This identifies the ScamperInst that the measurement should be executed on. If the ScamperCtrl object is operating with a single instance, then you do not have to pass an inst parameter – it will use that single instance. Otherwise, you have to identify the specific ScamperInst that the measurement should be executed on. The third common parameter is the userid parameter, which is an integer value that the caller can tag on the measurement for its own purpose. The value has to be able to be represented in an unsigned 32-bit integer, as that is what scamper uses internally.

do_trace(dst, confidence=None, dport=None, icmp_sum=None, firsthop=None, gaplimit=None, loops=None, hoplimit=None, pmtud=None, squeries=None, ptr=None, payload=None, method=None, attempts=None, all_attempts=None, rtr=None, sport=None, src=None, tos=None, wait_timeout=None, wait_probe=None, userid=None, inst=None, sync=False)¶

Schedule a traceroute measurement to the IP address identified by dst. The other parameters are optional; if they are not specified, then the scamper process that does the measurement will use its defaults.

The confidence parameter specifies a confidence level to reach before assuming all the interfaces have been observed that will reply at that hop, and can be either 95 or 99. sport and dport specify the (base) source and destination TCP/UDP ports to use in probes. icmp_sum specifies the checksum to use in the ICMP header for icmp-paris traceroute. firsthop specifies the TTL to start probing with. gaplimit specifies how many consecutive unresponsive hops before traceroute should stop. loops specifies the number of loops allowed before traceroute stops. pmtud, if True, instructs scamper to do Path-MTU probing of the path, to identify MTU changes in the path. squeries specifies the number of consecutive hops to probe before stopping to wait for a response. ptr, if True, instructs scamper to look up the name of any IP address observed in a response. payload specifies a bytes object with a payload to include in each probe. method specifies the type of probing strategy to use: icmp-paris, udp-paris, tcp, tck-ack, udp, or icmp. attempts specifies the number of attempts to try, per TTL; if all_attempts is True, then all attempts are sent, otherwise scamper stops after receiving a response for the hop. rtr specifies the IP address of the first hop router, overriding the system’s choice of router. src specifies the source IP address to use in probes. tos specifies the value to put in the 8-bit field previously known as IP Type of Service. wait_timeout specifies the length of time to wait for a response to a probe. wait_probe specifies the minimum length of time between consecutive probes.

The userid, inst, and sync parameters are documented above. This method will return a ScamperTrace object if sync is True, otherwise it will return a ScamperTask representing the scheduled measurement.

do_tracelb(dst, confidence=None, dport=None, firsthop=None, gaplimit=None, method=None, attempts=None, rtr=None, sport=None, tos=None, wait_timeout=None, wait_probe=None, userid=None, inst=None, sync=False)¶

Schedule an MDA (load-balancer) traceroute measurement to the IP address identified by dst. The other parameters are optional; if they are not specified, then the scamper process that does the measurement will use its defaults.

The confidence parameter specifies a confidence level to reach before assuming all the interfaces have been observed that will reply at that hop, and can be either 95 or 99. sport and dport specify the (base) source and destination TCP/UDP ports to use in probes. firsthop specifies the TTL to start probing with. gaplimit specifies how many consecutive unresponsive hops before traceroute should stop. method specifies the type of probing strategy to use: udp-dport, icmp-echo, udp-sport, tcp-sport, or tcp-ack-sport. attempts specifies the number of attempts to try, per TTL. ptr, if True, instructs scamper to look up the name of any IP address observed in a response. rtr specifies the IP address of the first hop router, overriding the system’s choice of router. tos specifies the value to put in the 8-bit field previously known as IP Type of Service. wait_timeout specifies the length of time to wait for a response to a probe. wait_probe specifies the minimum length of time between consecutive probes.

The userid, inst, and sync parameters are documented above. This method will return a ScamperTracelb object if sync is True, otherwise it will return a ScamperTask representing the scheduled measurement.

do_ping(dst, tcp_seq=None, tcp_ack=None, attempts=None, icmp_id=None, icmp_seq=None, icmp_sum=None, dport=None, sport=None, wait_probe=None, wait_timeout=None, tos=None, ttl=None, mtu=None, stop_count=None, method=None, payload=None, rtr=None, recordroute=None, size=None, src=None, userid=None, inst=None, sync=False)¶

Schedule a ping measurement to the IP address identified by dst. The other parameters are optional; if they are not specified, then the scamper process that does the measurement will use its defaults.

tcp_seq and tcp_ack specify the values to insert in the TCP sequence and acknowledgment fields. attempts specifies the number of probes to send. icmp_id, icmp_seq, and icmp_sum specify the ID, sequence, and checksum values to use in the ICMP header. sport and dport specify the source and destination port values for TCP and UDP probe methods. wait_probe and wait_timeout specify the length of time between probes, and how long to wait for a response to a probe. tos and ttl specify the type-of-service and TTL values to use in the IP header. mtu specifies a pseudo-MTU value to use for the measurement; if a response is larger, then scamper will send a packet too big to induce fragmentation in subsequent responses. stop_count specifies the number of probes that need a response before stopping. method specifies the probe strategy to use: icmp-echo, icmp-time, udp, udp-port, tcp-syn, tcp-syn-sport, tcp-synack, tcp-ack, tcp-ack-sport, or tcp-rst. payload specifies a bytes object with a payload to include in each probe. rtr specifies the IP address of the first hop router, overriding the system’s choice of router. recordroute, if True, includes an IP record-route option in probes. size specifies the size of probe packets to send. src specifies the source IP address to use in probes.

The userid, inst, and sync parameters are documented above. This method will return a ScamperPing object if sync is True.

do_dns(qname, server=None, qtype=None, attempts=None, rd=None, wait_timeout=None, userid=None, inst=None, sync=False)¶

Schedule a DNS measurement for the name identified by qname. The other parameters are optional; if they are not specified, then the scamper process that does the measurement will use its defaults.

server specifies the IP address of the DNS server to use. qtype specifies the type of query to make. attempts specifies the number of queries to send before stopping. rd, if True, signals to the DNS server that this is a recursive query. wait_timeout specifies the length of time to wait for a response.

The userid, inst, and sync parameters are documented above. This method will return a ScamperHost object if sync is True, otherwise it will return a ScamperTask representing the scheduled measurement.

do_http(dst, url, headers=None, insecure=False, limit_time=None, userid=None, inst=None, sync=False)¶

Schedule an HTTP GET to the IP address specified in dst using the URL specified in the url parameter. The other parameters are optional; if they are not specified, then the scamper process that does the measurement will use its defaults.

headers specifies a dictionary that maps header names to values to include in the GET request. insecure, if True, allows scamper to ignore TLS certificate errors with an HTTPS and proceed with the request. limit_time specifies the maximum length of time that the HTTP query may take.

The userid, inst, and sync parameters are documented above. This method will return a ScamperHttp object if sync is True, otherwise it will return a ScamperTask representing the scheduled measurement.

do_udpprobe(dst, dport, payload, inst=None, userid=None, sync=False)¶

Send a UDP probe to the IP address and port specified by dst and dport, with the specified payload.

The userid, inst, and sync parameters are documented above. This method will return a ScamperUdpprobe object if sync is True, otherwise it will return a ScamperTask representing the scheduled measurement.

do_ally(dst1, dst2, fudge=None, icmp_sum=None, dport=None, sport=None, method=None, attempts=None, wait_probe=None, wait_timeout=None, userid=None, inst=None, sync=False)¶

Schedule an Ally-style alias resolution measurement for the two IP addresses identified by dst1 and dst2. The other parameters are optional; if they are not specified, then the scamper process that does the measurement will use its defaults.

The fudge value specifies the maximum difference between IPID values to still consider as possibly from the same counter. icmp_sum specifies the ICMP checksum value to use in probes. sport and dport specify the (base) source and destination ports to use in probes. method specifies the probing strategy: udp, udp-dport, tcp-ack, tcp-ack-sport, tcp-syn-sport, or icmp-echo. attempts specifies the total number of probes to send in this measurement. wait_probe and wait_timeout specify the length of time between probes, and how long to wait for a response to a probe.