Technical metrics and evaluation methods for each project area, proposed in 2010:
|Architecture Component||Key Evaluation Metrics||Evaluation Method(s)|
|Routing and Data Delivery||FIB size, PIT size and lifetime, routing & Interest message overhead, successful path probability and length distribution, delay in finding data, optimal number and diversity of paths||testbed measurement, simulation, theoretical analysis|
|Hardware||FIB update delay, packet processing delay (including lookup delay)||testbed measurement|
|Caching||cache size, hit ratio as a function of content type||testbed measurement, simulation|
|Flow Control||interface throughput, link utilization||testbed measurement, simulation|
|Application support||ease of creating applications (e.g. closeness of mapping between application needs and network support), application-level data throughput||testbed measurement|
|Privacy||privacy preservation capability of TORNADO (NDN version of TOR)||testbed measurement|
|Data security||speed of generating and verifying signatures||testbed measurement|
|DDoS||percentage of requested data delivered to legitimate users||testbed measurement, simulation|
|Capacity and Traffic||total amount of information transported by the network in space and time (i.e. consumed entropy rate), traffic patterns compared with IP||testbed measurement, theoretical analysis|
David Clark's list of criteriaEvaluation may proceed in two steps. The first is to ask what each of the designs says about these points. The second and more challenging is to ask how one would compare those answers and decide that one was "better". Our field has few tools to do that.
- Availability and resilience
- Economic viability
- Better management
- Meet society's needs : (but, hmm, "The Internet was Drastically good at restricting choice to one QoS")
- Longevity : Extensible/flexible header, or fixed, foundational building block? [ e.g., jtw's definition of architecture is ``what is (intended to be) left after 30 years''].
- Support for tomorrow's computing
- Exploit tomorrow's networking
- Support tomorrow's applications
- Fit for purpose (it works...)
Other methods for evaluating architectures
- Evaluation by assessment of impact: how (and for whom) would world be different? app designers or end users?
- Evaluation by comparison: either via list of requirements or structured another way, e.g., by mechanism and subsystem
- Evaluation by simulation (since many features only show efficiency at scale)
- Theoretical evaluation: by derivation of optimal design, or by derivation on bounds and limits (if we can find any)
- Evaluation by "extreme use case", e.g., post-disaster operation (open question: What use cases are appropriate to demonstrate that the architectures will work in a real-world environment? examples of interest: critical infrastructure, enterprise content delivery network with embedded clouds, a civil crisis or public safety network, a complex network of things, large-scale interconnected cyber-physical systems.
- Evaluation by red team
- Evaluation by deployment and use, with caveat that implementation issues may not refect core design principles
Suggested next step for each project
a 10-page overview where is the archictecture now: lucid explanations of architecture. what have you learned, what was right / wrong, what have you had to add, take away. what you thought was critical but turned out not to be. e.g., clarify essential problems in NDN forwarding, and how how data structures and algorithms should be applied to solve them. separate architecture from mechanism; separate architecture from systems research