Project Summary

Project Title: Atomised Routing

Organization: CAIDA, NLnet Labs and Ripe NCC

Start Date: October 2002

End Date: October 2003 (tentative)

Principal Investigator: kc claffy and Andre Broido

Project URL: http://www.caida.org/funding/routing/atoms/

Overall Objective

In the atomised routing project we are researching and implementing modifications to BGP routing that aggregate prefixes into equivalence classes (policy atoms) based on common AS path from a given topological location. The motivation behind development of BGP atomization mechanisms is to achieve potential savings in computation and communication costs (by absorbing routing dynamics of prefixes into coarser grained atoms), as well as reduction in BGP table size (there will be far fewer atoms than prefixes). The project proposal can be found at http://www.caida.org/funding/routing/atoms/proposal/index.xml and http://www.caida.org/funding/routing/atoms/proposal.pdf.

Item: Refining and Releasing Atom Computation Scripts

Objective:

Documenting and releasing Andre Broido's Perl scripts for atoms computation.

Background:

The atoms computation scripts perform two tasks. They derive BGP statistics from the University of Oregon RouteViews `show ip bgp' tables, and they compute BGP policy atoms (groups of prefixes that share the same AS path in each BGP router). Earlier versions of the scripts were used for many of the results in papers authored by Andre Broido et al.:

• Analysis of RouteViews BGP data: policy atoms
• Complexity of global routing policies
• Internet Expansion, Refinement, and Churn

Results:

The bulk of the original scripts were rewritten for readability and easy usage, code reviewed, and source code and usage documentation were provided. The results have been posted on the Web and can be downloaded from http://www.caida.org/funding/routing/atoms/download/. Announcements were made to a number of mailing lists.

The output generated by the scripts include:

• Atoms: a list of computed atoms, a mapping of each prefix to its atom, and a distribution of prefix counts per atom.
• A rich set of statistics about ASes, prefixes, RouteViews peers and AS paths.
• Various AS graphs in text form, intended for further processing.
• The original BGP table dump in a standard, easy-to-parse format. Ambiguities and irrelevant information have been removed from this file, which will be beneficial for future scripts and programs.

Sample of Results:

A sample of the results of running the scripts on the RouteViews dump of 1 February 2003 (noon) is given below:

Number of prefixes	125857
Maximum number of prefixes per peer in /0-/24 range	119400
Number of prefixes shared by 35 peers	113343
Number of ASes with positive indegree	14754
Number of ASes with positive outdegree	2353
Number of BGP atoms	29059
Number of origin-declared atoms	21771 (derived)

The above results were computed over the 35 RouteViews peers that carry at least 114000 prefixes (with prefix length 24 or less). The full set of files for this run is also available. The number of origin-declared atoms is actually the number of distinct origin links (first link in the AS path) seen. Declared atoms are discussed later.

Current Plan:

In their current version the scripts compute `strict' atoms, that is, in strict accordance with the definition of atoms given above. It is our intention to include alternative definitions of atoms (such as `crown atoms' in Analysis of RouteViews BGP data: policy atoms or Complexity of global routing policies), which better correspond to the atom definition given in the project proposal.

The scripts produce many statistics and are written in Perl. Geoff Huston has written an optimised C program which specialises in the task of computing atoms alone. He intends to include atom statistics on http://bgp.potaroo.net/. In addition, a group at Tel Aviv University is researching BGP atoms and have written an independent implementation of atoms computation. Finally, an associate at CAIDA is writing a Python implementation of atoms computation.

Item: Implementation of atomised routing

Objective:

To design, implement and measure an atomised router, atomised routing and atomised forwarding.

Results: Identifying key issues

As part of the design of atomised routing, we have enumerated and elaborated a list of key issues that need to be resolved in order to come to a BGP protocol based on atoms. Each issue has different trade-offs in terms of performance and complexity. An early version of this list was sent to the atoms mailing list. Some of the more significant issues and trade-offs are listed below.

Issue: to declare or compute atoms

Atoms can be declared by origin ASes (ASes that originate prefixes), or they can be computed from BGP tables, e.g. using the scripts that were described earlier.

• Declared atoms require cooperation from origin ASes. Computed atoms on the other hand can be determined independently of origin ASes, and are therefore less intrusive.
• As mentioned in the proposal, it is as yet uncertain whether atoms computation can be done in a scalable way. Computation of atoms requires taking a global snapshot of a number of BGP routers. Declared atoms, on the other hand, can be introduced by each origin AS independently.
• Computed atoms reflect the actual routing state and behaviour of BGP routers. In contrast, declared atoms attempt to alter the behaviour of BGP routers; their intent is to make each BGP router apply the same policy to the prefixes in atom. An important issue for declared atoms therefore is that each BGP router does indeed comply with this behaviour. If some BGP routers apply the same policy to all prefixes in a declared atom but others do not, inconsistency may result, which may lead to forwarding loops and blackholing of traffic.

We are currently working on a possible solution that combines atoms computation with atoms declaration.

Issue: islands or scattered routers

There are two deployment scenarios: `islands' of atomised routers embedded in the Internet, or a scattering of isolated atomised routers across the Internet.

• Islands allow atomised forwarding (tunneling tagged packets across an island), as described in the project proposal. This in turn allows routers within the island to carry fewer destinations in their tables (one destination per atom, rather than per prefix).
• Scattered atomised routers appear to be more easily deployed, since they do not require a contiguous area of atomised routing.
• Keeping atomised routers consistent with non-atomised routers is a non-trivial issue. An island of atomised routers is a region within which consistency issues do not arise. Consistency is only an issue at the edge of the island, and therefore affects a relatively small number of atomised routers. Scattered atomised routers (effectively islands of size one) do not have this advantage.

Issue: absorbing instability

Atoms offer the potential of absorbing some instability of prefixes in much the same way that CIDR aggregation is able to absorb instability. We intend to exploit this property of atoms.

Results: extending the BGP protocol

The BGP protocol needs to be extended to carry both the distribution of atom contents (what prefixes are assigned to an atom), and the routing of atoms as a whole (through what paths can an atom be reached). For incremental deployment it is essential that atomised routing and forwarding be transparent to non-atomised BGP routers. For example, updates for atomised prefixes must be sent to non-atomised BGP routers so that they may continue to route these destinations. Avoiding inconsistency in BGP appears to be one of the main challenges here.

We have drafted two alternative extensions to the BGP protocol, which are summarised below. Both extensions are based on declared (rather than computed) atoms. The extensions can be used in a scenario of atomised routers scattered across the Internet or within an island of atomised routers. The extensions are as follows:

• In the first extension, the prefixes in an atom, as well as the atom itself (as represented by its atom ID), are routed as destinations in the BGP protocol. A BGP attribute containing the atom ID is attached to each of these. The attribute allows atomised routers to recognise what prefixes belong to the same atom. Routing computations are performed on the atom route and applied to each of the prefixes in the atom.
• In the second extension, an atom (as represented by its atom ID) is routed as a destination in the BGP protocol; however the prefixes in an atom are not routed as separate destinations (except where necessary to keep non-atomised routers up-to-date). Instead, a BGP attribute containing a list of prefixes is attached to the atom route.

When implemented in the context of an island, these schemes can be supplemented with an atomised forwarding component.

One important issue which has arisen is how an atomised router should resolve conflicting atoms definitions from different neighbours. For example, an atom A1 learned from neighbour N1 may overlap (have prefixes in common with) an atom A2 learned from neighbour N2. This situation may arise after a prefix moves from one atom to another and BGP has not yet converged for these destinations.

Results: implementation

We are currently implementing the first of the above schemes as modifications to the Zebra code base. At this point we are debugging an initial rough implementation.

There are very few open source router implementations, and Zebra appears to be the only reasonable choice for us to base an atoms implementation on. For example, GateD is no longer open source, and MRT is not actively maintained. However, the Zebra code base is rather unstructured and mostly undocumented, and some time has been spent understanding Zebra's BGP implementation. It seems that the community would be well served with an alternative open source router implementation.

Current Plan:

• To complete implementation of the above routing extensions. We expect to have that completed by April 2003.
• To implement atomised forwarding by modifying Zebra and a FreeBSD kernel by May 2003.
• To measure the performance in terms of BGP computation, BGP table size, BGP communication, and packet forwarding rate, relative to each other and to non-atomised routing. We have seeded a collaboration with Agilent through which to perform these measurements.
• We expect that our extensions, which are based on declared atoms, can be adapted to support computed atoms. In the future we will investigate modifications that support computed atoms.

Item: Feedback from community

Objective:

Obtain feedback about atomised protocols and implementation.

Results:

• We consulted and obtained valuable feedback from a number of experts in the area of routing at IETF-55, summarized at: http://www.caida.org/funding/routing/atoms/notes/ietf-comments.txt. This feedback from the operational community has influenced our thinking about how to progress, what to measure, etc. Specifically:
  • An important aspect of atoms is information hiding and reduction. Great emphasis was laid on the possibility of using atoms as a buffer against instability of prefixes.
  • We don't know how performance is hurt most, whether it is the number of prefixes, the amount of communication, or the dynamics of routing tables. We should simulate and measure to find out. We expect to do experimental testing in a lab collaboratively designed and built with Agilent in spring and summer 2003.
  • The need to consider business relationships among providers, customers and peers with respect to atoms has become clear.
  • Misconfiguration of declared atoms is another important issue to consider.
  • With regard to tunneling of IP packets through an atomised island, we received mixed reactions. Some believe it is too expensive; others believe that as a mechanism it is not unlike MPLS and an efficient implementation should therefore be feasible. In any case, using IP source routing (as in the proposal) is considered a bad idea.
  • We received an offer to help deploy atomised routing in a research network.
• A mailing list has been set up for the purpose of discussing atoms at: http://login.caida.org/mailman/listinfo/atoms. The response by subscribers of the list so far has been mild; folks are busy. We are considering (and open to suggestions on) how to encourage more interaction on the mailing list.

Current Plan:

We will try to further encourage feedback by posting a summary of current status of design and implementation on the atoms web pages as well as a list of issues that we need feedback on.

Considering the usefulness of the feedback we received at IETF-55, we intend to test our latest ideas about atomised routing at IETF-56 (March 2003 in San Francisco), and will be taking further advantage of exposure to the operational experts who have helped us thus far. We will provide a summary of these interactions by 1 April 2003.

Item: BGP Bibliography

Objective:

Provide the community with an online annotated list of BGP related research.

Results:

A BGP bibliography was created at http://www.caida.org/publications/bib/networking/. So far sixteen entries have been registered.

Current Plan:

We will continue adding papers to the bibliography, and kc will also hold a UCSD summer quarter reading seminar course that will involve reading 10 landmark papers in BGP research relevant to this project. Details forthcoming in a subsequent report.