Slide 1

cflowd and arts++
Cisco flow-export collection






dwm@caida.org

Slide 2

Background


host software for collecting Cisco version 5 flow-export data
aggregates data into tables for continuous collection of summary data in time-series
stores raw flow data in rotating log files
provides client/server collection of tabular data in time series

Slide 3

Changes to Upcoming Release


central collector included
uses arts++ package
arts++ adds signficant functionality

Slide 4

cflowd

Slide 5

flow export version 5


sends UDP packets to a specified host address and port number
each UDP packet contains a number of flow entries

Slide 6

flow-export packets

Slide 7

flow header

Slide 8

flow entry

Slide 9

cflowd aggregation


cflowd primarily designed to aggregate flow data into tabular data to be used for capacity planning
AS matrix, net matrix, port table and protocol table aggregation across all flows

Slide 10

Why so much aggregation?

data reduction
scale of intended use (backbone-wide)
Add a cflowd host, you may add it to central collection of tabular data.
unreliable transport from Cisco to cflowd encourages deploying multiple cflowd hosts; use them.
disk and bandwidth requirements for raw flow data in a backbone network.  Only transfer tabular data back to the central collector.

Slide 11

cfdcollect


permits collection of cflowd data at intervals
end result is time-series data for each of the tabular data types (AS matrix, net matrix, port table and protocol table)
stores data in ARTS files

Slide 12

centralized collection

Slide 13

arts++


C++ class library for subset of ARTS data
supports reading/writing of ARTS data via iostreams and UNIX file descriptors
supports simple time-domain aggregation for several data types
simple command-line utilities included for viewing ARTS data files and time domain aggregation

Slide 14

ARTS data files


efficient data archival (binary, simple size-reducing techniques)
data files are portable; always written in network byte order, the arts++ class library is the interface
extensible for additional data types
versioning of data objects for different storage formats (typically used for space/CPU tradeoff)

Slide 15

ARTS data handled by arts++


AS matrix (version 0)
net matrix (version 2)
port table (version 2)
protocol table (version 2)
forward IP path (version 0)

Slide 16

ARTS AS matrix (version 0)


counters for traffic (packets and bytes) from source ASes to destination ASes
sparse matrix, having only entries for which traffic information is stored

Slide 17

ARTS AS matrix example data

Slide 18

ARTS net matrix (version 2)


counters for traffic (packets and bytes) from source networks to destination networks
networks are identified by network number and netmask length
sparse matrix, having only entries for which traffic information is stored

Slide 19

ARTS net matrix example data

Slide 20

ARTS port table (version 2)


counters for input and output traffic (packets and bytes) versus transport layer port number
input counters represent traffic destined for the port while output counters represent traffic sourced from the port
table is sparse; there are no entries for ports on which no traffic was seen

Slide 21

ARTS port table example data

Slide 22

ARTS protocol table (version 2)


counters (packets and bytes) versus IP protocol (TCP, UDP, ICMP, IGMP, et. al.)
sparse table, there are no entries for protocols that were not seen in the measured traffic

Slide 23

ARTS protocol example data

Slide 24

ARTS IP forward path (version 0)


contains IP addresses of hops in forward path from a source to a destination
contains an RTT value for the source to destination
may be extended in the future to hold more information

Slide 25

ARTS forward IP path example data

Slide 26

Aggregation Utilities


Time domain aggregation:

artsasagg
artsnetagg
artsportagg
artsprotoagg

Slide 27

Simple display utilities


artsdump
artsases
artsnets
artsports
artsprotos

Slide 28

Future Tools


plotting utilities using XRT/PDS
utilities to generate data files for JClass Chart

Slide 29

Open Questions


what types of aggregation are useful to network service providers?
are there desired applications for flow-export outside of capacity planning and usage/billing?