Logo

FANTAIL API Documentation

FANTAIL API Documentation

Synopsis

The fantail-api script allows you to execute queries and analysis recipes via the FANTAIL API.

Installation

Put the fantail-api and fantail_utilx.py script files in the same directory anywhere in your PATH.

To run fantail-api, you need Python 3 and two Python packages, which you can install with pip:

    $ python3 -m pip install requests docopt

Usage Summary

General usage:

fantail-api [options] <subcommand> [parameter] …

Determining vantage point selection parameters:

fantail-api select-vp
fantail-api codes

Executing standalone queries:

fantail-api addr TARGET
fantail-api dest TARGET
fantail-api neigh TARGET …

Executing analysis recipes:

fantail-api recipe NAME   TARGET   [KEY_VALUE …]

Checking the status of queries and retrieving results:

fantail-api status RESULT_ID
fantail-api get RESULT_ID

Options Summary

Authentication:

--key STR    (set API key)
FANTAIL_API_KEY environmental variable

Vantage point selection:

-C CODES, --continent CODES    (list of continent codes)
-U CODES, --country CODES    (list of country codes)
-O CODES, --orgtype CODES    (list of organization types)

Match conditions:

-t TIMERANGE    (time range to search)
--status REASON    (only match traces with the given status)
--pathlen LENRANGE    (IP path length range to search)
--destrtt RTTRANGE    (destination RTT (ms) range to search)

Data processing:

--annotate NAMES    (list of annotations)
--transform NAME    (data transformation)

Output control:

-l NUM    (max number of traces to retrieve [default: 1000])
-J    (output in JSON where available)
-V    (verbose output)
--out PATH    (path for results of get subcommand)

Misc:

--url URL    (API base URL [default: https://fantail.caida.org/api])
--timeout NUM    (HTTP request timeout (secs) [default: 120])
-h, --help    (show usage summary)
--version    (show version)

Introduction

There are two major ways of executing queries with fantail-api. Use the low-level addr, dest, and neigh subcommands to execute standalone queries with maximum configurability. Use the high-level recipe subcommand to execute analysis recipes, which are predefined multi-step querying and processing pipelines with limited configurability, because most choices are purposely hard-coded by the recipe to provide a simple turn-key solution.

The addr, dest, and neigh subcommands accept the full set of available options, whereas the recipe subcommand accepts a subset of these options, as detailed in further sections.

FANTAIL executes queries in three major steps:

  1. Fetch matching traceroute paths,
  2. Annotate traceroute paths with one or more metadata, and
  3. Transform traceroute paths.

The fetch step is always a part of a query, but the annotation and transformation steps are optional and may be configured with command-line options, except in the case of the recipe subcommand, since recipes configure the annotation and transformation steps themselves.

New users should read the following sections in order to first learn about the most important command-line options, namely, ones for specifying authentication, vantage points, and the date range of data to query. Later sections then cover the remaining command-line options, how to execute standalone queries and recipes, and how to retrieve results.

Authentication

Authentication is via an API key. Either pass the key via the --key option, or set the FANTAIL_API_KEY environmental variable.

Specifying Vantage Points and Date Ranges

Traceroute data in FANTAIL is organized by vantage point and date/time, and the best way to limit the total amount of data FANTAIL has to consider during query execution, and thus speed up query execution, is to specify the vantage points and date range you are interested in. By doing so, you determine the base set of traceroute data for querying, an initial coarse filtering of data that can happen quickly because it’s done without regard for the contents of traceroute paths (such as the presence of specific hop addresses). You can then refine your query over this base set of traceroute data by specifying additional query options (to be discussed later) that match on the contents of traceroute paths.

Vantage Points

Vantage points (VPs) have four attributes: name/ID, continent, country, and organization type. You cannot specify VPs by name/ID. Instead, you must specify the continents, countries, and/or organization types that you want VPs to have, and FANTAIL will use the VPs that meet those criteria.

For a specified set of continents, countries, and organization types, a VP meets all criteria if:

VP is in one of the continents (if any), AND
VP is in one of the countries (if any), AND
VP is one of the organization types (if any).

For example, if you specify

  • continents { Africa, Asia, Europe } and
  • organization types { “business”, “residential” },

then only VPs that are

(located in Africa OR Asia OR Europe) AND
(have organization type “business” OR “residential”)

will match the criteria.

Use the following options to specify vantage points:

-C CODES, --continent CODES    (list of continent codes)
-U CODES, --country CODES    (list of country codes)
-O CODES, --orgtype CODES    (list of organization types)

To reduce typing, each option takes a short code; in particular, ISO standard two-letter continent and countries codes.

To see a list of accepted codes, execute the codes subcommand:

$ fantail-api codes
orgtypes:
        business
        commercial
        educational
        infrastructure
        research
        residential

continents:
        af = Africa
        as = Asia
        eu = Europe
        na = North America
        sa = South America
        oc = Oceania
        an = Antarctica

countries:
        af = Afghanistan
        ax = Åland Islands
        al = Albania
        dz = Algeria
        as = American Samoa
        ad = Andorra
        ao = Angola
...

For example,

-C af,as,eu -O business,residential

corresponds to

  • continents { Africa, Asia, Europe } and
  • organization types { “business”, “residential” }.

It’s good practice to check how many VPs match your criteria before executing a query and adjust your criteria if too few/many VPs match. If you execute the select-vp subcommand with the VP selection options, then fantail-api will list the total number of VPs matching all criteria as well as a breakdown of VPs per individual criteria.

For example, without options, select-vp will list the total number of available VPs and their breakdown by criteria:

$ fantail-api select-vp
139 matching VPs
12 in af = Africa
14 in as = Asia
45 in eu = Europe
51 in na = North America
8 in oc = Oceania
9 in sa = South America
3 in ar = Argentina
1 in at = Austria
2 in au = Australia
1 in ba = Bosnia and Herzegovina
...
41 in us = United States of America
2 in za = South Africa
1 in zm = Zambia
6 in business
10 in commercial
37 in educational
19 in infrastructure
17 in research
50 in residential

And for our running example, you’d get

$ fantail-api -C af,as,eu -O business,residential select-vp
24 matching VPs
4 in af = Africa
5 in as = Asia
15 in eu = Europe
3 in business
21 in residential

Some criteria may not yield any VPs:

$ fantail-api -C af select-vp
12 matching VPs
12 in af = Africa

$ fantail-api -O commercial select-vp
10 matching VPs
10 in commercial

$ fantail-api -C af -O commercial select-vp
0 matching VPs
0 in af = Africa
0 in commercial

Date Ranges

When you specify a date range, only traceroute paths with a starting timestamp that falls within the date range will match the query. A date range is specified in UTC.

Use the following option to specify a date/time range:

-t TIMERANGE

The TIMERANGE argument should consist of the starting and ending date/time values separated with a comma, as follows:

-t D1,D2

which means D1 <= time <= D2. A missing starting/ending value in TIMERANGE means the earliest/latest possible vaue, respectively. So, the following two forms

-t D1,
-t ,D2

mean time >= D1 and time <= D2.

Each component of a TIMERANGE can take one of the following forms:

  • Unix timestamp: 1430443968
  • year: 2015
  • year-month: 2015-01
  • year-month-day: 2015-01-02
  • year-month-day HH:MM:SS: "2015-01-02 03:04:05"

For example:

-t 1430440000,1430450000
-t 2015-03-04,
-t ,2016-11
-t "2015-01-02 03:04:05",1430443968

Executing Standalone Queries

Use the following subcommands to execute standalone queries that search for traceroutes based on user-provided target addresses (standalone means independent of a recipe):

fantail-api addr TARGET
fantail-api dest TARGET
fantail-api neigh TARGET …

Note, the neigh subcommand takes two or more TARGET arguments.

The TARGET argument should consist of one or more IPv4 addresses or prefixes separated with commas (no spaces are allowed in the argument); for example, 1.2.3.4,192.168.0.0/16. You can think of the TARGET argument as specifying a set of target addresses made up of the individually listed addresses and the addresses belonging to listed prefixes (for the above example, the set { 1.2.3.4, 192.168.0.0, … , 192.168.255.255 }).

In general, these subcommands search for traceroutes that contain at least one member of the set of target addresses; the subcommands differ in where or how the target address may appear in the traceroute path.

Specifically,

  • addr searches for traceroutes with the target address as a hop or a responding destination,
  • dest searches for traceroutes with the target address as the destination, regardless of whether the destination responded, and
  • neigh works similarly to addr, except a given traceroute must have at least one address from each TARGET argument (you can think of the addr subcommand as a degenerate case of neigh with just one TARGET argument).

The neigh subcommand doesn’t require target addresses to appear in adjacent hops or that the appearances are in the same order as the corresponding TARGET arguments.

For example:

  • fantail-api   addr   10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

    Searches for traceroutes with RFC1918 private addresses as hops.

  • fantail-api   neigh   4.0.0.0/9   206.126.236.0/22

    Searches for traceroutes that pass through both 4.0.0.0/9 (Level 3) and 206.126.236.0/22 (Equinix Internet exchange point).

Additional Match Conditions for Standalone Queries

In addition to matching traceroute paths based on IP addresses, FANTAIL supports matching on three additional conditions; namely,

--status REASON    (only match traces with the given status)
--pathlen LENRANGE    (IP path length range to search)
--destrtt RTTRANGE    (destination RTT (ms) range to search)

Measurement Status

Use --status to match traceroutes that stopped their measurement for one of the following reasons (REASON argument):

  • completed – traceroute finished successfully by getting a response from the destination,

  • gaplimit – traceroute stopped after 5 successive non-responding hops despite multiple attempts per non-responding hop (this often means that a firewall is blocking probes or that there’s no host at the destination address),

  • unreach – traceroute stopped on ICMP destination unreachable, and

  • loop – traceroute stopped due to the detection of a loop; that is, an address seen earlier in the traceroute path is seen again, except when repeated in adjacent hops (with some caveats).

Path length

Use --pathlen to match traceroutes with a path length that falls within the given length range. The LENRANGE argument should consist of the lower and upper bounds separated by a comma, as follows:

--pathlen L,U

which means L <= length <= U. A missing lower/upper value in LENRANGE means the lowest/highest possible vaue, respectively. So, the following two forms

--pathlen L,
--pathlen ,U

mean length >= L and length <= U.

Destination RTT

Use --destrtt to match traceroutes with an RTT, in ms, from a responding destination that falls within the given range. A traceroute that didn’t get a response from the destination (that is, didn’t have a completed status) will never match a --destrtt condition.

Like LENRANGE above, RTTRANGE should be a lower and upper bound on the RTT separated with a comma.

Data Processing for Standalone Queries

FANTAIL can perform two optional data processing steps on traceroute paths that match a standalone query.

--annotate NAMES    (list of annotations)
--transform NAME    (data transformation)

Use --annotate to add metadata from CAIDA datasets to traceroute paths. The NAMES argument should be a comma-separated list of one or more of the following values:

  • aliases – ITDK MIDAR aliases,
  • ases – AS assignments from ITDK bdrmapIT,
  • hostnames – IPv4 Routed /24 DNS Names Dataset, and
  • ixps – CAIDA Internet eXchange Points (IXPs) Dataset.

See the Traceroute Annotations section of the FANTAIL File Formats documentation for further details.

Use --transform to analyze or process traceroute paths into a different form. The NAME argument should be one of the following values:

  • hop-addrs – extract unique IP addresses, including the destination if it responded,
  • ip-links – extract unique IP links,
  • ip-paths – extract unique IP paths,
  • ip-rtts – calculate min, max, avg, stddev, and percentiles (25th, 50th, 75th, and 95th) of RTTs for each IP hop/destination per vantage point,
  • peering-links – extract unique AS peering links, and
  • router-graph – generate a router-level graph using alias resolution data.

See the FANTAIL File Formats documentation for further details about the algorithms and output formats.

Executing Analysis Recipes

Use the recipe subcommand to execute an analysis recipe, a predefined multi-step querying and processing pipeline.

fantail-api recipe NAME   TARGET   [KEY_VALUE …]

The NAME argument should be one of the following:

  • peering-links – extract unique AS peering links of the target AS, and
  • router-graph – generate a router-level graph of the target AS using alias resolution data.

For the currently supported recipes, the TARGET argument should be an AS number. The optional KEY_VALUE arguments provide a way to pass additional configuration information to recipes, but no current recipe has additional configurations.

These recipes work like the data transformations with the same name described in the [Data Processing for Standalone Queries] section, except these receipes only return the data (such as peering links) involving the target AS.

The recipe subcommand only supports the options described in the [Specifying Vantage Points and Date Ranges] section. In particular, the subcommand doesn’t support any of the options described in the [Additional Match Conditions for Standalone Queries] section or the [Data Processing for Standalone Queries] section, because a recipe has preconfigured, unchangeable values for these options.

Checking the Status of Queries and Retrieving Results

Execution of queries and recipes happens asynchronously on the FANTAIL server. When you run the addr, dest, neigh, or recipe subcommand, the fantail-api submits a job to the server and then prints the RESULT_ID, a number, for the job.

To check whether the results are ready, run

fantail-api status RESULT_ID

which will output a status of queued, inprogress, finished, or error along with various metadata and the error message, in the case of an error.

For example,

$ fantail-api -U au -t 2022, addr 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
result ID: 1
$ fantail-api status 1
status: inprogress
...
$ fantail-api status 1
status: finished
{
    "end_date": "",
    "limit": 1000,
    "method": "addr",
    "source": [
        "per-au",
        "per2-au"
    ],
    "start_date": "1640995200",
    "target": "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
    "target2": ""
}
submission date: Thu Jan  5 14:33:11 2023
completion date: Thu Jan  5 14:33:16 2023

When execution has finished, retrieve results with

fantail-api get RESULT_ID

For example,

$ fantail-api get 1
$ echo $?
0

On success, the get subcommand outputs nothing and has a success return code. By default, output is written to the file fantail-RESULT_ID.out in the current directory; for example, fantail-1.out.

For a standalone query without the --transform option, as in our example, the result will be a JSON file with one traceroute path per line. See the FANTAIL File Formats documentation.

Published