Basic CoralReef command usage

CoralReef applications whose names begin with "crl_" accept command line options of the form -C "coral_command", followed by the names of one or more CoralReef sources. Note that if the option contains spaces, it must be quoted from the shell. For example,
crl_trace -Cd=600 -C "comment site_name" /dev/point0


CoralReef applications operate on "sources" of network traffic. Simple sources include several types of live network interfaces/devices, and several types of tracefiles containing data recorded from one or more network interfaces/devices. The name of a source may be prefixed to indicate what type of source it is. If the name has no prefix, CoralReef will attempt to determine the type by the filename suffix (the suffixes ".enc" and ".gz" are skipped); if there is no suffix, CoralReef will attempt to determine the type by the file's device type or magic number.

CoralReef also recognizes "compound" sources, which consist of the concatenation of the data in multiple simple sources. On the command line of a crl_* application, a compound source is specified by surrounding a list of simple sources with "[" and "]". (There must be space between the brackets and the names in the list.) The files must be listed in time order. All simple sources that make up a compound source must be the same type and should not have prefixes. If a prefix is required, it should be placed before the "[", with no space. The following example has one source consisting of three dag files:

crl_flow -Ci=60 dag:[ trace1 trace2 trace3 ]

The types are:

Live sources

For all source types, bandwidth and physical options are useful only to tell CoralReef what to expect from the source, if CoralReef can not determine it from the source itself. The bandwidth and physical options never change the configuration of the source.

Trace file sources

For all file sources, the special filename "-" indicates that a tracefile should be read from stdin. If libcoral was compiled with libz, trace files in most formats may be gzipped, even on stdin. As of version 3.5, trace files are not limited to regular files; they may also be devices (e.g., /dev/fd/*) or FIFOs.

Configuration Commands

These commands can be used on the command line as the argument to a -C option, or in a configuration file. Valid commands are listed below, but not all commands are meaningful in all applications. For commands that take arguments, the arguments may follow '=' or whitespace.
# comment

Print the CoralReef package version and the current file format version.

Read CoralReef commands from filename. This is normally used on the command line, but can be used in a config file to chain config files.

Defines a string that will be written into the file header by the coral_write functions.

Defines the default iomode options for subsequent coral sources. On a live source, an iomode option specifies how much data to capture per packet, or other configuration. On a file source that does not contain the options used to record it (i.e., NLANR/MCI .crl, legacy DAG, and TSH formats), an iomode option can be used to tell CoralReef what to expect in the file. Multiple options may be separated by commas or whitespace. If a boolean option is preceded by "!", it will be disabled. Some applications ignore this command and use a fixed iomode. In those that don't, the default is usually "rx,first=48" (i.e., capture only the first cell of each AAL5 PDU). Note that not all sources support all options; see the section on sources above. Options:
source has N interfaces.
capture the first N bytes of layer 2 PDU (packet or frame). On ATM devices, N is rounded up to a multiple of 48 bytes (1 cell).
capture last cell of each ATM AAL5 PDU (packet).
capture all bytes of each packet (overrides first).
capture OAM/RM ATM cells.
capture idle ATM cells.
capture all ATM cells. Equivalent to "user,ctrl,idle".
allow captured records to have variable length, to reduce bus bandwidth use (DAG ERF only).
use file name as firmware image for FATM or POINT cards. The default is selected automatically based on the source type and iomode, and should usually not be changed. You should use this option only if you know what you're doing, and use the correct source type and iomode.
set a device to use bandwidth rate, if it is configurable;
set a device to use physical layer type type, if it is configurable. If this is not specified, CoralReef may infer it from the data link layer protocol defined by a "proto" command.
Set the data link layer protocol of the device. See the "proto", "allow", and "deny" commands for details.
enable echoing of received data (aka passthrough or loopback) on devices which support it.
allow receiving (this usually should not be changed on command line).
allow transmitting (this usually should not be changed on command line).
source contains IEEE 802.1Q VLAN. If you want to use a pcap filter, this option is necessary to work around a bug in libpcap (as of version 0.7.0-281, at least).
As of CoralReef version 3.5, these options are no longer supported, since DAG cards are expected to be pre-configured.

Defines a coral source sourcename, as described under Sources. There may be multiple "source" commands; each one defines a new CoralReef source with its own options. The options may be any of the options accepted by the "iomode" command above, and will override any options specified in earlier "iomode" commands. Options specified in a "source" command affect only the source defined in the same "source" command.

Tells libcoral to interpret traffic as data link protocol protocol.
proto and allow tell libcoral to accept traffic from subinterface subif, and optionally to interpret it as data link protocol protocol.

deny tells libcoral to discard traffic from subinterface subif. Protocol is accepted but ignored, to make it convenient to change a config file line from proto to deny and back without deleting protocol.

The proto, allow, and deny commands are collectively called "protocol rules". They are not valid in applications that read blocks (as opposed to cells or packets; i.e., crl_trace). When subif is an ATM vpi:vci, these commands specify RFC 1483/2684 VC Based Multiplexing. Subif should not be used with other types of interfaces (to filter by IEEE 802.1Q VLAN, use a '-Cfilter vlan vlan_id' command.)

On interfaces that have subinterfaces (i.e., ATM), packets are tested against the subinterface rules in the order the rules are defined, and the first matching rule is used. Packets that do not match any rule or match a rule without a specified protocol are assumed to have the interface's protocol.

Protocol is usually a layer 2 encapsulation. If null encapsulation (sometimes called "aal5mux" on ATM subinterfaces) is used, protocol may be a layer 3 protocol. Valid protocol names are listed below (although not all of them make sense in a protocol rule; some are used only for displaying information).

For ATM interfaces, a subinterface is specified as "vpi:vci". A field of subif will be interpreted as hex if it begins with "0x", otherwise octal if it begins with "0", otherwise decimal. A field of "*" will match all possible values.

For pcap interfaces, the link layer protocol is determined from the interface. For ATM interfaces, if no protocol rules are specified, the default link layer protocol for virtual channels 0:0 through 0:15 is UNKNOWN, for 0:16 is ILMI, and for all other virtual channels is ATM_RFC1483. For DAG POS interfaces, the default link layer protocol is CHDLC. For all other interface types, the default link layer protocol is UNKNOWN.

Example rules for an ATM network with signaling on VPVC 0:16, null-encapsulated ("aal5mux") IPv4 on VP 27, and RFC1483 LLC/SNAP on VP 42, and uninteresting traffic on all other VCs:
A final "proto=ATM_RFC1483" rule could have been given, but is not necessary, since ATM_RFC1483 is the default data link protocol for ATM devices. The "allow=42:*" rule could also have been written "proto=42:*=ATM_RFC1483"

In CoralReef applications that read packets, only packets that match the tcpdump/pcap expression will be read. If multiple filter expressions are specified (by multiple "filter" commands and/or by the application), they will be joined with "and". See the documentation for tcpdump for the syntax of expression. If pcap can not parse the layer 2 headers (e.g., MPLS), the filter may not work as expected; see ipfilter for a solution. Note: due to the design of libpcap and BPF, when reading a VLAN interface, you must specify the "vlan" iomode option or begin your expression with "vlan and", or operations on layer 3 and above will not work. If your filter does not need to analyze layer 2 headers, you can use ipfilter instead to avoid needing to specify "vlan". Also note, due to a bug in libpcap 0.6.2, it is not possible to apply a filter to VLAN and non-VLAN sources simultaneously with libpcap 0.6.2 (this was fixed in 0.7.0). You can also use "crl_to_pcap -r" to strip the link layer encapsulation from a VLAN source to make it a non-VLAN source.

Note that discarded packets still count towards duration and interval.

A "prefilter" is like a filter, except it is used only on ATM interfaces, on the first ATM cell, before AAL5 reassembly. Any packet whose first cell does not match the prefilter does not need to be reassembled, which can improve performance. However, a prefilter works correctly only if the first ATM cell contains enough information to apply the prefilter. It is up to the user to make sure the prefilter does not need information that does not fit in the first cell.

An "ipfilter" is like a filter, except that it uses CoralReef's parser to skip past layer 2 to find an IP packet. If no IP packet is found, the packet is discarded; otherwise, the pcap filter is tested starting at the IP header. This is useful when packets contain IP encapsulated in some layer 2 protocol that pcap filters can not parse, but CoralReef can, such as MPLS. Note: due to a bug in libpcap, the "ip" operator matches IPv6 packets in addition to IPv4, and the "ip6" operator matches IPv4 in addition to IPv6. (This is true through at least version 0.9.4 of libpcap; it may or may not be fixed in later versions.) To work around this, you may use "ip[0] & 0xf0 == 0x40" instead of "ip" to match only IPv4 packets, and "ip[0] & 0xf0 == 0x60" instead of "ip6" to match only IPv6 packets.

Anonymize layer 3 packets in packet reading applications. For IPv4 and IPv6 packets, this means anonymizing the IP addresses in the header, incrementally updating the IP, TCP, UDP, and/or ICMP checksums, and recursing into the payload if it contains nested anonymizable packets or truncating the payload if it does not. Examples of nested anonymizable packets include IPv6 in IPv4, IPv4 in IPv6, the original IP packet contained in an ICMP error response, the gateway in an ICMP REDIRECT, and intermediate addresses in IPv4 Record Route or IPv6 ROUTING. All other layer 3 packets have their layer 3 header truncated to 0 bytes (by default). If no anonymizable layer 3 header is found, the packet is truncated at the end of the header of the highest recognizable layer (by default). Layer 2 addresses (e.g. Ethernet) are not anonymized. Anonymization is applied after filtering.

Protocol selection options:
All subsequent options apply to both IPv4 and IPv6 packets (default).
All subsequent options apply to IPv4 packets only.
All subsequent options apply to IPv6 packets only.

Algorithm selection options:
Do not anonymize IP addresses (default).
Set the bits of IP addresses to zero. (To be useful, this should probably be combined with one or more of the optional parameters.)
Do prefix-preserving anonymization of IP addresses with the Crypto-PAn algorithm. If bits is less than the full address length, the trailing bits will be zeroed. For IPv6, a reasonable choice might be "cryptopan,bits=64": this will preserve prefix relationships in the network part of the address, but obliterate the host part.

IP-specific options:
anonymize IP source address only
anonymize IP destination address only
anonymize only the first N bits of IP addresses; defaults to all bits.
anonymize only the last N bits of IP addresses
do not anonymize multicast addresses ( and ff00::/16). (If used with cryptopan, there is no guarantee that other anonymized addresses will not collide with multicast addresses.)
If an IPv6 address contains an embedded IPv4 address, then do not apply normal IPv6 anonymization, but instead apply IPv4 anonymization to the embedded IPv4 address(es). Coral recognizes embeded IPv4 addresses in the following IPv6 address schemes: IPv4-mapped (::ffff:0:0/96), SIIT (::ffff:0:0:0/96), Teredo (2001:0000::/32), 6to4 (2002::/16), 6over4 (fe80::/96), and ISATAP (fe80::5efe:0:0/96). Additionally, for 6to4 addresses, the last 80 bits are zeroed. (If used with cryptopan, there is no guarantee that other anonymized addresses will not collide with these special addresses.)
(cryptopan only) filename is the name of a file containing at least 32 bytes used to initialize the anonymizer. If you use the same key with two different coral sources, Crypto-PAn will produce the same prefix and address mappings in both. We recommend using different keys for IPv4 and IPv6. Default: /dev/random.

Non-IP options (setting of ip, ipv4, or ipv6 is irrelevant):
Do not truncate non-anonymizable packets. Note that this may leave ARP or other packets containing IP addresses that will not be anonymized.

For example, to apply cryptopan to IPv4 source addresses and all IPv6 addresses, with different keys for each: "anon=cryptopan,ipv4,src,keyfile=./ip4.key,ipv6,keyfile=./ip6.key".

Another example: zero the low 80 bits of normal IPv6 addresses and the low 11 bits of IPv4 addresses, including those embedded within IPv6 addresses, but leave IPv6 multicast addresses alone: "anon=zero,ipv4,bits=-11,ipv6,bits=-80,embed,keepmcast".

Note that prior to version 3.8.5, when anonymization was enabled, libcoral would discard packets that did not have a recognizable layer 3 header and packets whose outermost layer 3 header was not anonymizable. Now, those packets are not discarded, but are truncated at the beginning of the unanonymizable part (unless "notrunc" is specified). If you wish to discard non-IP packets, use "filter='not ip'" or similar option.

Equivalent to "anon=ipv4,cryptopan[,keyfile=filename],ipv6,zero" (for backward compatibility).

In applications that read packets, stop reading after N packets (including packets skipped by -Cskippackets). 0 indicates no limit. Default is 0.

In applications that read packets, skip the first N packets. Default is 0.

In applications that read cells, stop reading after N cells. 0 indicates no limit. Default is 0.

In applications that read blocks or cells, stop reading after N blocks. 0 indicates no limit. Default is 0.

In some CoralReef applications, this determines how long to run; 0 usually indicates unlimited runtime. Time may be in seconds, or may be in "hours:minutes[:seconds]" format. E.g., "-Cd=3600" and "-Cd=1:00" both set duration to 1 hour. Default is 0.

Discard all packets whose timestamp is earlier than time. Time may be in seconds, or may be in "hours:minutes[:seconds]" format. Note that discarded packets still count towards duration and interval.

Discard all packets whose timestamp is later than time. Time may be in seconds, or may be in "hours:minutes[:seconds]" format.

In CoralReef applications that perform periodic operations, this determines how long that period should be. Time may be in seconds or in "hours:minutes[:seconds]" format; seconds may contain a decimal point followed by up to 6 digits. The value 0.0 usually indicates that no periodic operation should be done.

Enable interval alignment. The end time of the first interval will be rounded down to the nearest integer multiple of the interval length. For example, if an application is started at 13:27:52 with options "-Ci=00:05:00 -Cai", the first interval ends at 13:30:00 (and since it began at 13:27:52, the length of the first interval is only 0:02:08, not 0:05:00). Subsequent intervals will thus start on rounded boundaries: 13:30:00, 13:35:00, and so on. Interval rounding makes the most sense when the interval length is evenly divisible into 1 hour. Note that this option does not cause the duration to be rounded.

Disable interval alignment. This is the default for most applications.

If time is nonzero, then approximately every time seconds, coral will report any dropped packets on live pcap interfaces to the error file. The dropinterval is independent of the normal interval and alignint.

Sets the verbosity level of libcoral and possibly the whole program. Level 0 is error messages only; level 1 includes warnings; level 2 and higher include additional information. Default is 1.

Enable normalization of all timestamps to the unix epoch, if possible. (This is the default for most applications.)

Disable time normalization. Timestamps may then be relative to the unix epoch or to the time the interface was started. Applications that read packets, sort by time, or use interval or duration may ignore this option and force normalization when reading from multiple interfaces that were not started at the same time.

Enables gzip compression of trace files written by crl_trace, crl_to_pcap, crl_to_dag, or any other application that uses coral_write() or coral_rf_ogz_write(). The level parameter can be 0 (no compression), or between 1 (fastest) and 9 (best compression). The default compression level is 1 to make it practical for use on live sources. (In versions of CoralReef before 3.5, the default level was 6).
Enables time sorting of packets or cells in applications that don't sort by default.

By default, libcoral reading functions return an error when they encounter a timestamp error they can not correct in a source. With this option enabled, libcoral will continue reading normally, but beware that intervals and duration may not work correctly.

Error/diagnostic output of libcoral and possibly the whole program will be written to filename. Default is stderr. In applications that use rotating files, the error file will also rotate if filename contains "%", but any error messages generated before the first rotation will go to stderr.
The "source", "iomode", and "filter" commands are cumulative; for any other repeated commands, only the last one given is effective. So, for example, if file "coral.cfg" contains "verbosity=2", and the command line options are "-Cverbosity=0 -Cconfig=coral.cfg", then the verbosity level will be 2; but if the command line options are reversed, the verbosity will be 0.

Some other common options available on some applications are:

Output to file filename. If filename ends in ".gz", or the -Cgzip option was specified, the file will be gzipped. The recommended suffix for (uncompressed) CoralReef files is ".crl". The special filename "-" indicates standard output.
When reading packets, do not read partial packets
When reading packets, do read partial packets (enabled by default)
Display a summary of the options available in this application. (Note: in some shells, you may need to type -\? or '-?').

Rotating files

Additionally, some applications have the ability to put timestamps in the names of their output and error files and to rotate these files occasionally. When rotation is used, the filenames should contain % specifiers for timestamp formatting. In most applications, the timestamp used to format filename is not the current time, but the timestamp of the current point in the input data; this allows useful filenames to be generated even when the input is from a pre-recorded file. The % specifiers are any of those allowed by strftime(), plus:
number of (whole) seconds since 1970-01-01 00:00:00 UTC
fractional part of seconds (6 decimal places)
equivalent to %Y-%m-%d
rotation counter
Except for %f, all specifiers may contain printf-style modifiers (e.g., "%03i"). Except for %s, all specifiers are formatted in the local timezone according to the TZ environment variable; to get UTC instead, set TZ="UTC" in the environment. The filename "-" means stdout, and disables rotation. Note that if the % specifiers are not sufficient to make the rotating filename unique with each rotation, most applications will force them to be unique by implicitly appending "%i".

Rotated files can be read by coral applications using the compound source feature, by enclosing the list of generated files in "[" and "]". If the name of the rotating file is chosen so that shell globbing sorts the names in temporal order, you can later use a convenient file glob to generate the compound source list on the command line. For example, if you took two sets of simultaneous traces, one with the rotating name "in-%Y%m%d-%H%M%S.dag" and the other with "out-%Y%m%d-%H%M%S.dag", they could later be read by another application like this:

crl_app ... [ in-20040921-*.dag ] [ out-20040921-*.dag ]


When specifying a protocol on the command line or configuration file, only the name is needed (e.g., PPP). In the C API, identifiers are formed by concatenating "CORAL_", prefix, "_", and name (e.g., CORAL_DLT_PPP). In the perl API, identifiers are formed by concatenating the prefix, "_", and name, in the Coral:: namespace (e.g., $Coral::DLT_PPP). In the table below, the "use" column contains "Y" if coral can parse the protocol and/or "P" if the protocol makes sense to use in a configuration protocol rule.

layer1 prefix name description use standards
1 PHY ATM Asynchronous Transfer Mode (The default link layer protocol for virtual channels 0:0 through 0:15 is UNKNOWN, for 0:16 is ILMI, and for all other virtual channels is ATM_RFC1483.) Y af-bici-0013.003
1 PHY POS Packet Over SONET (proto must also be specified; there is no default) -
2 DLT ATM_AAL5 ATM Adaptation Layer type 5 Y
2 DLT ATM_RFC14832 Multiprotocol Encapsulation over AAL5 (aka "aal5snap" on Cisco routers). Recognized protocols include IPv4, IPv6, ARP, PPP, and Bridged IEEE 802.3. YP RFC 1483, 2684
2 DLT LANE_IEEE8023 LAN Emulation for IEEE 802.3/Ethernet, over AAL5. (LAN Emulation for IEEE 802.5 is not yet implemented.) YP af-lane-0084.000
2 DLT ILMI Integrated Local Management Interface, over AAL5. YP af-ilmi-0065.000
2 DLT ETHER DIX Ethernet (also accepts IEEE 802.3), including IEEE 802.1Q VLAN YP
2 DLT IEEE802 IEEE 802.3 (also accepts DIX Ethernet), including IEEE 802.1Q VLAN YP IEEE 802.3, 802.2; RFC 1042
2 DLT IEEE8021D IEEE 802.1D bridge spanning tree IEEE 802.1D
2 DLT CHDLC Cisco HDLC, over POS YP RFC 1547 (4.3.1)
2 DLT MPoFR Multiprotocol over Frame Relay (also, Frame over SONET) YP RFC 1490, 2427
2 DLT UoPOS Unknown protocol over PoS - libcoral will attempt to parse this as CHDLC, PPPoHDLC, or MPoFR YP
2 DLT PPP Point-to-Point Protocol YP RFC 1661, 2364
2 DLT PPPoHDLC PPP over HDLC Y RFC 1662, 2615
2 DLT PPPoED PPP over Ethernet, discovery stage Y RFC 2516
2 DLT PPPoES PPP over Ethernet, session stage Y RFC 2516
2 DLT BRIDGED_LAN Bridged IEEE 802.3/Ethernet (over PPP) Y RFC 1638 (4.2)
2 DLT GIGX DEC Gigaswitch encapsulation (?), over AAL5 (no validation, assumed to contain ATM_RFC1483) YP
2 DLT MPLS Multi-Protocol Label Switching (may contain IPv4, IPv6, or ETHER) Y RFC 3032
2 DLT NULL BSD "null" YP
2 DLT LOOP OpenBSD "loopback" YP
2 DLT LINUX_SLL Linux "cooked" encapsulation YP
3 NETPROTO IPv4 or IP Internet Protocol, version 4 YP RFC 791, 3168
3 NETPROTO IPv6 Internet Protocol, version 6 YP RFC 2460, 3168
3 NETPROTO RAW_IP Internet Protocol (version determined from packet header) YP
3 NETPROTO ARP Internet Address Resolution Protocol (Ethernet and ATM) YP RFC 826, 1293, 1577, 2225
3 NETPROTO AARP AppleTalk Address Resolution Protocol YP
3 PROTO LCP PPP Link Control Protocol RFC 1661
4 IPPROTO TCP Transport Control Protocol, over IP Y RFC 793, 1323, 2018, 3168
4 IPPROTO UDP User Datagram Protocol, over IP Y RFC 768
4 IPPROTO ICMP Internet Control Message Protocol, over IPv4 Y RFC 792, 1192, 4884, 4950
4 IPPROTO ICMP6 Internet Control Message Protocol, over IPv6 Y RFC 2463
4 IPPROTO HOPOPTS3 IPv6 Hop-by-Hop Options Y RFC 2460
4 IPPROTO ROUTING3 IPv6 Routing Header Y RFC 2460
4 IPPROTO FRAGMENT3 IPv6 Fragment Header Y RFC 2460
4 IPPROTO ESP3 IPv4/IPv6 Encap Security Payload Y RFC 2406
4 IPPROTO AH3 IPv4/IPv6 Authentication Header Y RFC 2402
4 IPPROTO DSTOPTS3 IPv6 Destination Options Y RFC 2460
4 IPPROTO EGP Exterior Gateway Protocol RFC 888
4 IPPROTO MTP Multicast Transport Protocol
4 IPPROTO ENCAP Encapsulation Header RFC 1241
4 IPPROTO PIM Protocol Independent Multicast
4 IPPROTO SCTP Stream Control Transmission Protocol
4 IPPROTO GRE Generic Routing Encapsulation, version 0 Y RFC 2784
5 PROTO SNMP Simple Network Management Protocol Y RFC 1157; af-ilmi-0065.000
5 PROTO DNS Domain Name Service Y RFC 1035, 1876, 2136, 2671

1The division into OSI layers is crude, since many are actually sublayers (especially within layer 2).
2To specify RFC 1483 VC Based Multiplexing, use the proto configuration command.
3These are part of IP, but it is convenient to treat them as being encapsulated in IP or each other.

We expect to support more protocols in the future, depending on demand.