Overview

Packets captured by the UCSD network telescope infrastructure are written to pcap trace files and archived to disk for later analysis.

Definition

In STARDUST, pcap data is the raw traffic data collected from the network telescope. Each pcap file contains 1 hour of data and is typically over 100 GB large.

Pcap data from the last 30 days is stored in the telescope-ucsdnt-pcap-live container in the Swift object store. Older trace files, however, are rotated out and moved into a separate archive where a different process is used to access the data.

Properties

Packets contained within the pcap traces are unanonymized and not truncated in any way. They should exactly match what was observed on the wire at the telescope capture point.

User Guide

Recent pcaps (last 30 days)

The most recently generated trace files for the UCSD network telescope typically cover the last 30 days of telescope traffic and can be found in the telescope-ucsdnt-pcap-live container in the Swift object store.

Example

user@vm001:~$ tracepktdump pcapfile:swift://telescope-ucsdnt-pcap-live/datasource=ucsd-nt/year=2020/month=10/day=31/hour=22/ucsd-nt.1604181600.pcap.gz | head

Sat Oct 31 22:00:00 2020
 Capture: Packet Length: 60/64 Direction Value: -1
 Ethernet: Dest: 3c:fd:fe:19:d8:00 Source: 00:de:fb:ba:06:c7 Ethertype: 0x0800
 IP: Header Len 20 Ver 4 DSCP 00 ECN 0 Total Length 40
 IP: Id 54321 Fragoff 0
 IP: TTL 241 Proto 6 (tcp) Checksum 46313
 IP: Source 45.153.203.175 Destination 44.28.27.80
 TCP: Source 43922 Dest 82
 TCP: Seq 2846108233

Archived pcaps (available upon request)

Files can be brought back from the archive and into a container upon request, but this will not be an instantaneous process and will require manual intervention from a STARDUST administrator. Additionally, space on the Swift object store is limited so the amount of archived data that can be made available at any one time will be restricted. Please bear these limitations in mind when making a request for archived pcap data.

Reading/Processing Trace Files

Libtrace is the recommended method for reading and processing the pcap trace files, as it natively supports reading compressed pcaps through the Swift API.

Alternatively, if you have your own non-libtrace code that you want to run against the traces (e.g. something written using libpcap), you can use the wandiocat tool to stream the pcaps directly from Swift into your program (assuming it accepts stdin as an input source). Wandiocat installation instructions are documented, however, it should be installed on any STARDUST VM or container by default.

Please do not download the pcap files to your VM / container because the files are large and your disk space allocation is relatively small. Use libtrace or wandiocat to stream the data to you from the Swift object store instead.