Named Data Networking(NDN): Evaluation

Since the software implements components of an architecture, computational performance metrics of project software components running on a testbed, or in some cases using simulation, offer some objective measure against which to compare architectures (see below). But at FIA PI meetings, researchers have engaged in more interdisciplinarly conversations, recognizing that evaluating the releative merits of different future Internet architectures is not a purely technical exercise. At the first PI meeting, David Clark introduced a list of aspirations of a future architecture (see below), and used three PI meetings to intensely focus on describing the architectures with just a single aspiration (or two) in mind. We got through the first four topics on his list which led into an exploration of case studies in which to evaluate each architecture's efficacy. Eight external experts joined the October 2012 PI meeting in DC, observing that the conversation about evaluating architectures would benefit from more time, structure, and diversity of interests represented. The March 2013 PI meeting will provide an opportunity to accomplish all three. The solicitation for a follow-on program to FIA is due out in February, and will provide additional framing for this conversation. This page summarizes proposed suggestions related to evaluation.

Technical metrics and evaluation methods for each project area, proposed in 2010:

Architecture Component	Key Evaluation Metrics	Evaluation Method(s)
Routing and Data Delivery	FIB size, PIT size and lifetime, routing & Interest message overhead, successful path probability and length distribution, delay in finding data, optimal number and diversity of paths	testbed measurement, simulation, theoretical analysis
Hardware	FIB update delay, packet processing delay (including lookup delay)	testbed measurement
Caching	cache size, hit ratio as a function of content type	testbed measurement, simulation
Flow Control	interface throughput, link utilization	testbed measurement, simulation
Application support	ease of creating applications (e.g. closeness of mapping between application needs and network support[1]), application-level data throughput	testbed measurement
Privacy	privacy preservation capability of TORNADO (NDN version of TOR)	testbed measurement
Data security	speed of generating and verifying signatures	testbed measurement
DDoS	percentage of requested data delivered to legitimate users	testbed measurement, simulation
Capacity and Traffic	total amount of information transported by the network in space and time (i.e. consumed entropy rate), traffic patterns compared with IP	testbed measurement, theoretical analysis

David Clark's list of criteria

Evaluation may proceed in two steps. The first is to ask what each of the designs says about these points. The second and more challenging is to ask how one would compare those answers and decide that one was "better". Our field has few tools to do that.

Security
Availability and resilience
Economic viability
Better management
Meet society's needs : (but, hmm, "The Internet was Drastically good at restricting choice to one QoS")
Longevity : Extensible/flexible header, or fixed, foundational building block? [ e.g., jtw's definition of architecture is ``what is (intended to be) left after 30 years''].
Support for tomorrow's computing
Exploit tomorrow's networking
Support tomorrow's applications
Fit for purpose (it works...)

Other methods for evaluating architectures

Evaluation by assessment of impact: how (and for whom) would world be different? app designers or end users?
Evaluation by comparison: either via list of requirements or structured another way, e.g., by mechanism and subsystem
Evaluation by simulation (since many features only show efficiency at scale)
Theoretical evaluation: by derivation of optimal design, or by derivation on bounds and limits (if we can find any)
Evaluation by "extreme use case", e.g., post-disaster operation (open question: What use cases are appropriate to demonstrate that the architectures will work in a real-world environment? examples of interest: critical infrastructure, enterprise content delivery network with embedded clouds, a civil crisis or public safety network, a complex network of things, large-scale interconnected cyber-physical systems.
Evaluation by red team
Evaluation by deployment and use, with caveat that implementation issues may not refect core design principles

Suggested next step for each project

a 10-page overview where is the archictecture now: lucid explanations of architecture. what have you learned, what was right / wrong, what have you had to add, take away. what you thought was critical but turned out not to be. e.g., clarify essential problems in NDN forwarding, and how how data structures and algorithms should be applied to solve them. separate architecture from mechanism; separate architecture from systems research