Hackathon Dates: September 23 (Tuesday), 2025
Place: Room 408, San Diego Supercomputer Center, UCSD Campus, La Jolla, CA

Background

We hosted another Ark-focused hackathon the day before the main GMI-AIMS-6 workshop (September 24-26), focused on team coding challenges that leverage the new scamper libraries, and the Ark infrastructure.

We also hosted hands-on training on Thurs and Friday the 25-26th, focused on the UCSD Network Telescope.

Required reading for all hackathon participants:

• Documentation: Getting Started: Ark Integrated Active Measurement Programming Environment
• Paper: An Integrated Active Measurement Programming Environment
• Doc: Scamper Python module documentation
• Blog: Towards a Domain Specific Language for Internet Active Measurement
• Blog: Developing active Internet measurement software locally to run on Ark
• Tutorial: Internet Yellow Pages (IYP) Tutorial
• Slides: A Python-based Integrated Active Measurement Programming Environment

Hackathon Chairs:

• Matthew Luckie (CAIDA/UC San Diego)
• kc claffy (CAIDA/UC San Diego)

Scamper Orientation

scamper’s Python module

CAIDA’s scamper python module allows programmers to drive scamper Internet measurements across CAIDA’s Ark. The scamper tool is a packet probing utility used for active measurements of network characteristics like latency, traceroute, and packet loss. CAIDA’s Archipelago (Ark) platform is a global network measurement infrastructure consisting of strategically located vantage points that collect and analyze Internet data.

Running code

- Request access by signing the CAIDA Computer Facilities Usage Agreement
- Hostname provided to hackathon participants - The mux socket is located at /run/ark/mux

Background reading

- Full Documentation - full details on architecture, API, and 4 examples
- Paper on the scamper python module: An Integrated Active Measurement Programming Environment
more details on architecture and 3 examples. Related slidedeck is also available.

Example

The following example implements the well-known shortest-ping measurement technique, which conducts delay measurements to an IP address from a distributed set of vantage points, and reports the shortest of all the observed delays with the name of the vantage point that observed the delay. More details can be found in scamper Python module documentation.

ark-mux\> python3 simple-ping.py /run/ark/mux 192.172.226.122

import sys
from datetime import timedelta
from scamper import ScamperCtrl

if len(sys.argv) != 3:
  print("usage: shortest-ping.py $mux $ip")
  sys.exit(-1)

with ScamperCtrl(mux=sys.argv[1]) as ctrl:
  ctrl.add_vps(ctrl.vps())
  for i in ctrl.instances():
    ctrl.do_ping(sys.argv[2], inst=i)

  min_rtt = None
  min_vp = None
  for o in ctrl.responses(timeout=timedelta(seconds=10)):
    if o.min_rtt is not None and (min_rtt is None or min_rtt > o.min_rtt):
      min_rtt = o.min_rtt
      min_vp = o.inst

  if min_rtt is not None:
    print(f"{min_vp.name} {(min_rtt.total_seconds()*1000):.1f} ms")
  else:
    print(f"no responses for {sys.argv[2]}")

Hackathon projects

Develop tooling to analyze/visualize scamper results using ClickHouse and Grafana

Background: Scamper natively supports persisting results in warts files and provides various utilities to read warts files and produce a different output format, such as JSON (sc_warts2json) and text (sc_warts2text). ClickHouse is a popular, high-performance alternative to traditional time series databases, and allows analytical queries over large volumes of data (architecture overview).

Mission: The goal of this hackathon project is to develop tooling to load results from warts files into a ClickHouse database to allow for analytics using SQL queries (part I). A second objective is to visualize results using Grafana, for measurement types that allow for this (part II).

Intended Outcomes:

1. ClickHouse CREATE TABLE statements for supported scamper measurement types, with appropriate engine, table partitioning and primary index.
2. Document instructions to load results data into clickhouse database “indirectly” with existing utilities (e.g., piping sc_warts2json).
3. Ideally, develop tooling to the same effect (e.g., create a sc_warts2clickhouse utility and/or Python script that uses ScamperFile and CHI bindings).
4. Create Grafana dashboards to visualize select measurement result (aggregates), for measurement types that can be visualized with Grafana
   • Examples:
     1. Show time series of (avg) RTT, for a selected Ark VP and target
     2. Show nr. of unique NS records and/or unique /24s in (NS) address records for a selected popular domain name over time (see RFC2182 and Allman’s Comments on DNS robustness)
     3. Explore other Grafana visualizations (geomap, …)
5. (dashboards and underlying queries can be exported)
6. (optionally) sensible secondary indices for tables in clickhouse (see here)
7. (optionally) Docker compose scripts for ClickHouse and Grafana deployment
8. (optionally) add additional data to clickhouse for measurement enrichment (e.g., geolocation of hops, etc.)

Shepherds: Mattijs Jonker (U Twente)

Team Members: Jiayi Liu (), Nikolas Gauder (), Ricky Mok (CAIDA)

Tasks:

1. Docker compose script for Clickhouse and Grafana deployment [Jiayi]
2. Initial Clickhouse table definitions [Jiayi]
3. Create warts2clickhouse tool with ping, traceroute support [Jiayi]
4. Setup caida/github repo for project [Ricky]
5. Cloud deployment for demo [Ricky]
6. Create basic/example warts files for various measurement types [Nikolas]
7. Organize repo [Mattijs]
8. Add dns support to warts2clickhouse [Nikolas]
9. Brainstorm about case study [all]
10. Script and cron recurring ping+dns measurement [mattijs]
11. Script and cron recurring tracert measurement [mattijs]
12. Import longitudinal data [Ricky]
13. Create demo Grafana dashboard [Ricky]

Kafka output option for Scamper

Background: Scamper supports outputting results in various formats and persisting them to warts files. Apache Kafka is a highly popular open-source event streaming platform that is widely used in data processing pipelines.

Mission: The objective of this hackathon project is to devise a Kafka output option for Scamper so that results can be produced into Kafka.

Intended Outcomes:

1. A Scamper option to produce results to Kafka as well as the associated configuration settings (e.g., broker, topic, …) with support for basic authentication (i.e., SASL/PLAIN, SASL/SCRAM, and PLAIN as well as SSL support).
2. (alternatively or complementarily) Python boilerplate to wrap around ScamperCtrl and send event-driven results to Kafka upon receipt. This can leverage the confluent-kafka-python or kafka-python libraries.
3. (optionally) optimize the message output for size. For example, consider devising simple Avro schemas for common Scamper measurement types and produce results in avro-formatted messages rather than, e.g., JSON or CSV.

Shepherds: Mattijs Jonker (U Twente)

Create topology map of where such multi-homed ASes connect to the public internet

Background: multi-homed ASes: ASes like Microsoft, Cloudflare, Google, have a global presence with their own private backbone. Such networks connect to the wider Internet at edges (where traffic ingresses/egresses the private backbone)

Mission: using traceroute we have observed you can detect, enumerate, and geolocate these edges (similar to ANRW paper on anycast traceroute from previous Ark workshop)

Intended Outcome: map of where mult-homed ASes connect, identify routing strategies for various network (e.g., cold/hot potato routing, prefixes announced only in EU for GDPR, …) are valuable as a significant chunks of Internet traffic are from/to these ASes.

Shepherd: Raffaele Sommese (U Twente, remotely)

Team Members:

1. Florian Dekinder (University of Liège)
2. Ben Du (UCLA)

Country-in-the-middle measurements

Background: Within the OpenINTEL project we collect domain names from Certificate Transparency logs and make available weekly lists of (apex) domain names for 300+ country-code TLDs (see OpenINTEL data). Although CT logs do not account for all names in a zone, in prior work we’ve shown that CT logs account for a sizable part (~60% on average). Some zones feature extended TLD labels that are dedicated to government use, such as .gov.<cctld> and .gv.<cctld>. Additional labels and/or government domains can be learned by looking at the data of https://www.govdirectory.org/, which crowdsources domain names of government services, among others, for approximately 40 countries.

We call a country-in-the-middle (CITM) a country on the geographic path between a source and destination, where that country is neither the source nor destination country. Our ongoing work involves identifying CITMs to government websites of 11 countries around the world. This work relies heavily on manual analysis but finds some interesting results, so one potential hackathon idea is to develop a more automated system (other ideas are welcome).

See current paper status here: https://alishaukani.com/papers/CITM.pdf

Mission: Build a more automated system to monitor CITMs to government websites for longitudinal analysis.

1. Find government web sites hosted in cloud: 65K from CT logs (207 countries)) , 25K from govdirectory (39 countries)
   1. 6.2K are within cloud IP ranges in caida’s cloud prefix data set
2. Use scamper from Ark nodes to web sites
3. BdrMapIt traceroutes
4. Geolocate IP addresses
5. Visualize!

Shepherd: Alisha Ukani (UC San Diego)

Team Members:

1. Jarrett Huddleston (Johns Hopkins University)

Investigating anycast prefixes in R&E routing

Background: R&E institutions generally prefer R&E routes over commodity routes. This means that anycast routes that make their way into R&E routing may be chosen over a commodity route from a nearby peer, defeating the goal of anycast routing to direct traffic to a nearby instance. Generally, the anycast route should not be in R&E routing.

Mission: Using a list of R&E routes that are anycast, and a list of Ark VPs that are in R&E networks, determine the extra distance that an R&E institution may be travelling when reaching an anycast service. You can use ping and traceroute to characterize this. The following API endpoints will be useful:

1. https://rootbeer.testing.ns.internet2.edu/re-info/anycast/all
2. https://rootbeer.testing.ns.internet2.edu/re-info/origins

Intended Outcomes:

1. A dashboard that identifies problematic routes, who is announcing them, and how much effect these have in R&E.

Shepherds: Steven Wallace (Internet2), Matthew Luckie (CAIDA/UC San Diego)

Team Members:

1. Karl Newell (Internet2)
2. Jeff Bartig (Internet2)
3. Steven Wallace (Internet2)

Mapping hospital-related network IP ranges (IYP:Hospital)

Background: To map outages affecting specific organizations (e.g., Hospitals), we first need to infer the network ranges belonging to such organizations. Using hospital domains as a case study, can you utilize existing tools, platforms, and datasets to map the ranges of hospital networks? We have around 4000+ hospital domains across the US hospitals and clinics to start with (JAMA Paper for inspiration).

Mission: To map hospital network ranges, and monitor their availability over time (e.g., Hospitals might lose connectivity during targeted outages, e.g., power cuts, ransomware, etc)

Intended outcomes:

1. Find ASes that are health-related (check their ASdb categories, IYP categories, BGP.Tools categories, Borges?, As2org, AS names, etc..); Can this be refined?
2. Can we discover ranges which aren’t owned, but assigned to hospitals?
3. Can we discover cloud ranges which are used by hospitals?

Shepherds: Sumanth Rao (UC San Diego), Ye Shu

Team Members:

1. Yongzhe Xu (Virginia Tech)
2. Esteban Carisimo (Northwestern University)
3. Alexander Männel (TU Dresden)

Does the observed dual-stack addresses from vpn users actually belong to the same physical server?

Background: VPNs host dual-stack service. But do they really host dual-stacks on the same machine?

Mission: To find out whether it’s the same machine or not.

Intended outcomes: Look for unique characteristics of paired addresses (do they use the same public key? Anything like that?)

Team Members:

1. Yejin Cho (University of Southern California)
2. Kyle Trinh (CAIDA)
3. Max Gao (UC San Diego)

Livestreaming Aggregated Telescope Traffic Events to External Applications

Background: CAIDA can forward the live traffic received by the telescope to a VM from where telescope data consumers can then parse it. There is currently no public tooling to transform this traffic into a signal to for example move into kafka and perform some follow-up operation.

Mission: Create an application that allows for live parsing of the incoming traffic by BPF filter and converts it into a (customizable) JSON message that can be sent off via Kafka or websocket.

Intended Outcomes:

1. Revisit current tooling for using live traffic
2. Create a POC tool that turns the traffic into a signal after filtering
3. Use Hilby as a POC to visualize live incoming traffic or where traffic hits within the telescope space.

Shepherd: Alexander Männel (TU Dresden)

Scamper MCP

Background: Model Context Protocol (MCP) servers are used by Large Language Models (LLMs) to interact with external data sources. MCP servers allow interacting with data using natural language.

Mission: Implement a Scamper MCP server.

Intended Outcomes: Scamper MCP server.

Shepherd: Karl Newell (Internet2)

Telescope Tutorial/Experimentation

Identify targeted scannings to SDSC networks using “greynet” network telescope

Background: As a part of CAIDA’s STARNOVA project), we deployed a traffic aggregator to capture ingress traffic toward the network and broadcast addresses of subnets in the SDSC production network.

Mission: We have been collecting the traffic for 6 months. Our goal is to analyze the traffic and identify cyber threats to SDSC.

Approach: Compare SDSC greynet data with darknet (UCSD-NT). Back-probe the source with Ark.

Intended outcomes:

1. Identify scanners that possibly evade the detection of darknet
2. Identify threats targeted to the SDSC networks

Shepherd: Ricky Mok (CAIDA/UC San Diego)

Team Members:

1. Alexander Männel (TU Dresden)

Multishark (MULTISHARK)

(proposed, but not worked on)

Background: Tshark (the command line version of wireshark) cannot handle the large amounts of packets collected by the telescope directly. With a small C/C++ frontend, the packets from one pcap file could be forwarded to multiple tshark processes to both add “multiprocessing” support, as well as allow for restarts of tshark instances when they break.

Missions: Use libpcap to hack multithreading support in front of tshark for large amounts of data

Intended Outcomes:

1. Build the frontend, with forwarding of all tshark cli options
2. Test and benchmark the solution.

Shepherd: Alexander Männel (TU Dresden)

Telescope Report Page

(proposed, but not worked on)

Background: The UCSD-NT suffers from a set of operational issues that can impact research done in the timeframe they occur. Researchers using telescope data should be made aware of such events so timeframes can be adjusted and impact can be gauged.

Mission: Allow user to specify time window, generate summary of telescope metadata, e.g., size, disruptions and events.

Intended Outcomes:

1. Decide what data is relevant for researchers.
2. Build a webpage that presents the monitoring data from the telescope in an easy to digest way.
3. Add the state of the address space with individual available IPs to the status page.

Shepherd: Alexander Männel (TU Dresden)

Ideas for next hackathon

Measuring adoption of Serve Stale (RFC 8767)

Background: Serve Stale is a mechanism that enables DNS recursive resolvers to provide an answer to a client for expired cached records, in case the authoritative nameservers of the cached record domains are expired. Serve Stale is widely adopted on the Internet by several large recursive resolvers, yet no large-scale characterization of the adoption of these RFC has been done in the wild.

Mission: Leverage Ark nodes to identify instances of Serving Stale across recursive resolves of the nodes (and potentially, of different vantage points of recursive resolvers).

Raffaele already set up the authoritative nameserver with a domain name with the command to disrupt the connection and log the queries

Intended Outcomes:

1. A list of recursive resolvers that adopt serve-stale.
2. Investigating parameters and settings of the adopted implementations of serve stale in the wild. See: https://indico.dns-oarc.net/event/52/contributions/1151/attachments/1104/2292/Thinking%20About%20Serve%20Stale%20-%20OARC44.pdf

Shepherd: Raffaele Sommese (U Twente)