john,

geez, when you asked whether non-u.s. countries are building infrastructure to 'go around' the united states, i did not respond with "But economics also plays a role." the words you put right before my name in your article. What I said to you is that all the folks I know building network infrastructure outside the United States are doing it for economic reasons, per steve gibbard's article on "smart growth", and because one thing we know about network economics is that anyone who can afford to build their own infrastructure, does (which i also pointed you to). And specifically not because they are trying to avoid U.S. surveillance. The idea of U.S. hegemony over data flow is silly -- if the CIA is depending on it, we are doomed. Do you always insert sinister undertones even when simple economics explains the data?

k

As part of our DHS-funded cybersecurity project on Internet topology mapping with CAIDA's new Archipelago measurement infrastructure, DHS program manager Doug Maughan required a representative of each R&D project to attend a marketing workshop at SRI for some intense training on how to communicate the value of our specific projects to potential customers or sponsors. It was a 2-day format condensed from a typically week-long workshop based on (president of SRI) Curtis Carlson's book on the discipline of innovation. I went in to the workshop somewhat skeptical it would be useful. However, I recognize I have weak marketing skills since the scientist in me always wants to point out the dozen caveats of anything I'm presenting before I focus on the contributions. So I acknowledged I was an ideal candidate for the workshop.

The workshop was much better than I expected, mainly because the leaders, Laszlo Gyorffy and Gary Bridges, were outstanding at their jobs. (Doug apparently warned them I was going to be the person in the workshop most difficult to impress -- which I admit is probably true.) The goal of the workshop was to teach us the "NABC" approach to presenting our work -- the process of effectively communicating the need our project is addressing, the approach our project takes, the benefits our results offer, and the competing alternatives. Over the course of the two days, each of the ten attendees had to give a one-minute elevator pitch, a three-minute value proposition, and a five-minute business plan (day 2), punctuated by thoughtful structured feedback from the other nine attendees, the organizers, and Doug. We learned just as much about how to give useful feedback to others as we did about how to integrate feedback from others into our next revision.

I wholeheartedly agree with co-attendee Dale Peterson's assessment: It's hard to do the workshop justice. The pace was definitely agile -- a friend commented, "They're giving you two hours to write a business plan? That's insane, a business plan takes at least 2 full days to write." to which Gary responded, "Look, we just want to get you out of here pointing east. We don't care how far you get."

And easterly pointing we left, having visibly improved our presentations over the course of the two days, learned how to give more useful feedback to colleagues, and had a great time doing so. Even more benefit goes to taxpayers, who are more likely to see return on DHS technology investments if the funded investigators are comfortable explaining the value of their work to non-technologists. I wish I had gotten this training 20 years ago, in grad school, but better late than never. Kudos to Doug Maughan and DHS's Science and Technology (S&T) Directorate for their vision and execution.

[f.d.: S&T helps fund caida projects to (1) improve topology measurements and (2) increase data sharing, this year. -k.]

In April 2007 I was invited to David Isenberg's Freedom to Connect (F2C) conference to participate on a panel about Yochai Benkler's new book, Wealth of Networks (amazon, pdf chapters). In Wealth of Networks, Yochai first observes that two phenomena -- communication and computation -- are becoming affordable and ubiquitous at the same time that they are each becoming fundamental as input as well as output to our economic systems. He then provides empirical evidence [wikipedia] that this ubiquitous availability of information technology (communication and computational resources, or in math speak, links and nodes) among actors enables forms of collaboration so enormously effective as to offer an alternative to traditional models of production, i.e., market-based or government-backed systems.

Yochai emphasizes that highly distributed non-proprietary production is unlikely to -- nor should it -- displace market production. On the contrary, he argues that coexistence of the two modes will increase individual freedom, which is the goal of markets as well as democracy. Cooperative economic systems have always co-existed with competitive economic systems, and will continue to do so. But ubiquitous connectivity induces an inflection point on the economic landscape because it upsets the dominance of competitive, proprietary production of knowledge and culture, in terms of both economic return and individual autonomy. While property and markets enhance freedom along some dimensions, they constrain freedom in non-market contexts, often in disguise. So Yochai repeatedly reminds us to evaluate systems (whether political, economic, or technology) by gauging the extent to which a system allows individual can be the author of his or her life. Yochai thus offers an illuminating reality check regarding the impact of the Internet on our understanding of our economic selves. Poetically phrased in an amazon review, "If Marx and his Communist Manifesto were the tipping point for communism, this book is the tipping point for communal moral capitalism." The following is a more coherent version of my remarks at F2C following his excellent talk about the book.

I was happy to see Yochai's book come out because a few years ago I read his 2001 white paper, "Property, Commons, and the First Amendment: Toward a Core Common Infrastructure" at a time when I was struggling to identify ways that academics could help study if not help solve problems of the Internet. (Back before the Internet drove the phone system to bankruptcy, Bell Labs did this sort of thing for the country, and the world.) I had recently surveyed a few dozen engineers building and operating Internet infrastructure, asking "What are your top problems with operating and engineering the network?" I got over a hundred answers which fell along four dimensions: security, scalability, sustainability, and stewardship. Moreover, virtually all of their unsolved problems, even those seemingly mostly technical, were stalled due to economic or policy limitations, rather than anything related to technology. (Well, of course, they were engineers -- they solved the technology problems..)

Since CAIDA's research agenda has focused on the analysis of empirical data on operational networks, I spent more time navigating economic and policy issues than was good for me. So my first reaction to Yochai's idea to build a communication commons was "Great! A relevant public network to study! How can we build that?" So I read the whole book.

Consistent with the argument that ubiquitous communication infrastructure allows individual users to collaborate on world-improving projects, Yochai's appealing image of this new commons infrastructure includes operating with no institutional or financial support from the federal government: a we-the-people-who-build-mesh-networks-for-fun produce a public network commons, owned by "us". It sounds elegantly congruent with the Internet's architectural roots: end site control (aka freedom). Unfortunately, neither the technology nor the economics of today's infrastructure could support his utopian ideal. First, we lack the technology needed for mesh networks to operate without a backbone. That is, we don't know to route a billion (or even ten million) nodes without some required structure in the middle to achieve acceptable routing performance. In fact, the current Internet protocol does not even have enough addresses to build a ubiquitous network. Unfortunately, the replacement protocol (IPv6), which does have enough addresses but uses the same unscalable routing architecture, is not gaining a lot of traction in the current competitive ecosystem, where profits for innovation -- such as needed upgrades to addressing and routing architectures -- are driven to zero by design. Yochai's dream has a lot of reality in its way.

Yochai does admit that his conclusions about the relative power of cooperative production are not obviously applicable to the communications infrastructure itself, since communications devices are rival goods (aka atoms), leaving less certainty that a commons will deliver the required resources. He leaves open the possibility of technology changes, e.g., wireless mesh networking advances, that could increase the relative performance of commons-oriented communications policies from the perspective of individual autonomy. In the meantime, he enthusiastically focuses on the fact that for non-rival information goods which serve as both inputs and outputs to production processes, a commons model provides substantially greater security through an increasingly robust source of new information inputs and greater autonomy regarding their use. No argument there.

Another thing I liked about Yochai's book was his empirical grounding of arguments, as rare in the field of communications policy as it is in network science due to the difficulties in data acquisition. Yochai's measurement problem is as formidable as that of network scientists, since he has few analytic tools to determine that cooperative means of production outperform other means for science, news, journalism, entertainment, education, and religion. Like security and sustainability, we haven't even really agreed on what we're supposed to be measuring. Would our bank accounts be safer if linux had 95% of the desktop market? Will people get bored with Wikipedia updating?

But the lack of good data to resolve such debates only makes Yochai's case stronger, because uncertainty regarding which model is better for a given product renders essential that the playing field is level, rather than tilted away from the emerging mode of production by natural incumbent control of the competitive landscape. Eloquently paraphrased by one of my favorite government program managers, "So that means, either the feds should be explicitly promoting open source with tax dollars, or at least get out of its way."

Yochai compellingly explains why he wants to see a core commons infrastructure. It turns out that the academic sector has a surplus of what is needed to build such a commons (bandwidth, switching, students), since they currently have a dearth of what a commons could provide: data to support a field of network science. A commons project would not only facilitate this trade, but leave enough institutional space for experimentation with the social and economic practices of networked information production that Yochai argues is essential to the health of its institutional ecology. Yochai has an agenda, to be sure, but the scientific agenda is "to be sure" as well, so I'm a huge fan of Yochai's outlook.

"About 30 years ago there was much talk that geologists ought only to observe and not theorise; and I well remember saying that at this rate a man might as well go into a gravel pit and count the pebbles and describe the colors. How odd it is that anyone should not see that all observation must be for or against some view if it is to be of any service." -- Charles Darwin to W.W. Bates on Nov. 22, 1860

(gathered earlier this year upon a student's request)

1. Abatte, Janet. Inventing the Internet. 2000.
2. Benkler, Yochai. The Wealth of Networks. 2006.
3. Benkler, Yochai. Freedom in the COMMONS: Towards a Political Economy of Information., Duke Law Journal. 2003.
4. Brin, David. Transparent Society. 1999.
5. Cooper, Mark. Open Architecture as Communications Policy. 2004
6. Dodd, Annabelle. The Essential Guide to Telecommunications. 2005.
7. Dyer-Witheford, Nick. Cyber-Marx. 1999.
8. Ferguson, Charles H. The Broadband Problem: Anatomy of a Market Failure and a Policy Dilemma. 2004.
9. Horwitz, Robert B. The Irony of Regulatory Reform: The Deregulation of American Telecommunications. 1991.
10. John, Richard. Spreading the News. 1998.
11. Klinenberg, Eric. Fighting for Air. 2007.
12. Kushnick, Bruce. $200 Billion Broadband Scandal. 2006.
13. Mueller, Milton. Ruling the Root. 2002. (and Dave Crocker's review)
14. National Academies Press. Funding a Revolution: Government Support for Computing Research. 1999.
15. Noam, Eli. Beyond Liberalization II: "The Impending Doom of Common Carriage". 1994.
16. Nuechterlein and Weiser. Digital Crossroads: American Telecommunications Policy in the Internet Age. 2005.
17. Poole, Hilary, et al.. History of the Internet. 1999.
18. Roughgarden, Tim. "Selfish Routing and the Price of Anarchy". 2006.
19. Slotten, Hugh. Radio and Television Regulation. 2000.
20. Wilson, James Q. Bureaucracy: What Government Agencies Do and Why They Do It. 1991.

The Internet, an undergrad project by UCSD graphic art student Jennifer Hsu, while at CAIDA internship this year. (She's now in design school in NYC.)

[Jump to a Top Ten item: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10]
[Originally written as a series of blog entries, this document was later converted to a booklet/pamphlet, see  "Top Ten Things Lawyers Should Know About the Internet"]

#10: Moreover, even in the dim light of the underattended interdisciplinary research into the network, the available data implies clear directions for solutions, all of which cross policy-technology boundaries.

1. We can learn from our mistakes. The false assumption that competing members of a profit-maximizing ecosystem will cooperate toward architectural innovations not in their short-term interest is remarkably consistent across failed attempts to solve major problems of the Internet (e.g., ATM, multicast, routing security, IPv6, DNSSEC, QOS). Engineers have made valiant efforts to provide architectural solutions to security and scalability problems, providing vivid illustrations of how the computational thinking approach, embracing modularization and separation of issues, can fail to account for how tightly linked the technology, economic, and social dimensions of the problems are. As the Internet becomes the substrate underlying our professional, personal and political lives, we must recognize the links within and across its four biggest problems: (1) the fundamentally insecure software ecosystem, (2) the fundamentally unscalable routing and addressing architecture, (3) the fundamentally unsustainable economic architecture, and (4) a stewardship model broken along so many dimensions that solving, or even studying, the first three problems is no one's responsibility. Expecting the private sector to navigate these dimensions (security, scalability, sustainability, and stewardship) while subject to relentless pressure to minimize costs is a recipe for failure; even public-private partnerships are not free of these pressures. Furthermore, since all four dimensions transcend the jurisdiction of any sovereign government, we also cannot expect any solution that emphasizes national boundaries.
2. While competing in the middle prohibits architectural innovation, cooperating at the edge seems to be a common ingredient of the most successful innovations on the Internet, including the web and search engines, VOIP, Linux, Wikipedia, Ebay, the blogosphere and other social networks. Ubiquitous connectivity is transforming economic conditions, supporting collaborations among individuals that achieve more efficient means of production and consumption than either government programs or competitive markets have achieved. This transformation leaves some incongruity about the current economic architecture for the Internet, which has a deeply embedded preference for markets and private sector control of communications infrastructure as well as information. The extremely dynamic and unpredictable structure, usage, and growth of the Internet does not reduce the necessity of regulation to well-functioning markets; on the contrary, its elusive nature is what makes transparent and accountable experimentation so necessary.
3. What we believe about the infrastructure influences our technology and policy decisions. The current barriers to data access leave us without any mechanism to verify claims or weed out false beliefs about the infrastructure, including the increasing suspicion that the majority of Internet traffic represents illegal activity. Copyright infringement, only one example, may be so rampant as to be economically unviable to prevent, but without an objective look at how the network is used, we are subject to vain attempts to criminalize typical network usage rather than updating the laws to accomplish their intended purpose in light of technological developments. Ironically, traffic measurement undertaken by law enforcement for national security purposes and attempted by scientific researchers is also arguably illegal under current anachronistic legislation. Again, our choice is to cripple socially important goals -- law enforcement and scientific Internet research -- or update the relevant communications privacy (ECPA) legislation.
4. Public investment in knowledge production, including science and medical research, gains enormously from universal connectivity, offering distribution of resulting products to all taxpayers at zero marginal cost. The same reasoning reduces the justification for strong intellectual property systems, since they require expensive technology to prevent networks from doing what networks do naturally: share data. It is thus in interest of taxpayers for governments to promote and sometimes directly fund universal deployment of network infrastructure. More generally, government needs to prevent monopoly control over essential resources, mandate collection of traffic reports from ISPs to validate their claims, be a better role model for operational security, and coordinate the development of a roadmap for Internet security similar to that of the energy sector (DHS is working on this last one).
5. Scientific researchers are in a difficult position, trying to do science without data, but they are in a position to make progress, with the help of a few good legal experts. They (we) could propose a list of the most important Internet research questions/problems to investigate, such as the ongoing discrepancies between supposedly scientific studies, and suggest what data is needed to investigate them. The academic community could even use existing assets such as their own underutilized backbone to mitigate the data dearth, by incenting measurement data out of cooperating networks in exchange for network bandwidth. In the process they could help local communities experiment with and measure performance, cost, and efficiency of alternative network ownership models. Internet2 should also work with researchers and their institutional review boards (IRBs) at member universities to assist researchers in developing privacy-respecting network analysis technologies and data handling policies, so that the organization can share more data from its research backbone with scientific researchers.
6. The FCC is not exempt from the facts either -- the agency should be pursuing empirically grounded validation of the claimed efficiency of its own policies, even if it requires trading temporary spectrum unlicensing as an experiment to gather realistic baseline data on wireless network behavior to policymakers. The academic community could even help design such a network, geared toward public safety objectives and supporting scientific research balanced carefully against individual privacy. Such a trade seems less extreme an idea in light of the failure of the D-block auction, and the FCC admission that economic conditions make it a bad time to try to auction it now. Reforming our policy for this spectrum could achieve efficiency, access, public safety, and network science objectives at least cost to taxpayers.

   We can have facts without thinking but we cannot have thinking without facts. -- John Dewey

[Jump to a Top Ten item: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10]

#9: The news is not all bad: there is a reason everyone wants to be connected to all the world's knowledge -- as well as each other -- besides its status as the most powerful complex system ever created by man. The Internet's practical promise for individual freedom, democratic engagement, and economic empowerment, is also unparalleled. This promise is sufficient inspiration for an open, technically literate conversation about how to invest in technologies and policies to support articulated social objectives.

1. David Clark's conclusion that the federally funded network research community's "real accomplishment was not in computing but in connecting people" captures a century of thought. Although the openness of the architecture is the root of its many vulnerabilities, it was also the aspect that allowed enough self-organizing momentum to grow the network as fast as it did. The results are noisy, the journey messy, the future uncertain, the most pessimistic scenarios ominous. But the positive effects are also incalculable, particularly the potential for an unprecedented increase in individual freedom, the often deemphasized, yet primary, social objective of both democracy and markets.
2. The p2p file-sharing phenomenon, and more recently the user-generated video sharing phenomenon, are finally shedding some light on the inconvenient truth: we have not yet demonstrated a sustainable competitive model for moving raw bits around. Not that we excel at competitive models for moving things around over large distances to almost anywhere. Witness railroads, water, electricty, highways, postal service, telephony. Soon, airlines. The economics clearly need some sunlight. And the p2p debate will require some.
3. As with most infrastructure issues, the U.S. federal government is slow to respond regarding a national broadband strategy. But the USG is investing resources and regulatory attention to help foster global Internet growth, including: encouraging IPv6 deployment to mitigate the coming address space crunch; improving the security of the naming system with community-developed standards for authenticated DNS responses; and, in partnership with industry and academia, developing a roadmap for federal research and development in cybersecurity and information assurance. (Yes, the emphasis is on security and sustainability issues, but that's where federal investment is today.)
4. With infrastructure, progressive movement tends to begin at the state and local levels as governments experiment with alternative ownership models for provisioning Internet infrastructure via public-private partnerships. Local experimentation is critical, and eye-opening: after a decade of pay-per-minute hotspots, airports are realizing that free (as in beer) wifi access appeals to visitors and residents.
5. The OECD now considers the Internet relevant to its mission, and is issuing balanced recommendations based on its best available data, which they forcefully admit is problematic. In their recent ministerial meeting on the future of the Internet, they committed to "improving statistical indicators to measure access and use of the Internet..in order to provide more reliable data and analysis." Only in the U.S. do policymakers believe that OECD rankings arelying.
6. There are many educated people speaking out on the topic of informing policy based on what we know, and reserving judgment elsewhere. (Recommended thinkers.) There are evolutionary lessons and insights to glean from other networked fields facing similar problems, e.g., semantic web in big pharma and efficient routing as well as lessons to draw from ideas we have tried that have not worked yet, such as public catalogs or open commerce in network data. There's plenty of work to do, but there's no shortage of qualified people.
7. Authors and journalists have captured and interpreted history, and academic researchers have done their share of capturing and interpreting the history of communications and its implications for the Internet. There is detailed understanding of the history of many aspects of the Internet, including how pieces of the co-evolving complex systems of technology, economics, and regulation fit together.
8. Relatively few government-funded researchers, led by U.S. federal agency ARPA, supported by strong regulatory protection for innovation built the Internet in an amazingly short time relative to the history of communications. Within twenty years the new ecosystem fatally threatened the old. The obvious response by the incumbent carriers was to manipulate the regulatory architecture away from the line-sharing that made innovations such as the Internet possible. No surprise there, these same carriers fought innovation last century too, including the Internet. Regulating protection of innovation at the edge is neither new nor somehow obviated by the technological developments of the Internet. On the contrary, the technological ability to innovate at the edge of the Internet is easy to remove in the middle by a network owner. So as with the rest of history of telecom, and as with other social goals such as universal access, it will largely be a matter of pointing legislatures to results achieved from other policies.

But, important as these problems are, they were not the main point. The main point of the book is to see these human constructions as systems, not as collections of individuals or representatives of ideologies. From our opening accident with the coffeepot and job interview through the exotics of space, weapons, and microbiology, the theme has been that it is the way the parts fit together, interact, that is important. The dangerous accidents liein the system, not in the components. The nature of the transformation process eludes the capacities of any human system we can tolerate in the case of nuclear power and weapons; the air transport system works well -- diverse interests and technological changes support one another; we may worry much about the DNA system with its unregulated reward structure, less about chemical plants; and though the processes are less difficult and dangerous in mining and marine transport, we find the system of each is an unfortunate concatenation of diverse interests at cross-purposes. These systems are human constructions, whether designed by engineers and corporate presidents, or the result of unplanned, unwitting, crescive, slowly evolving human attempts to cope. Either way they are very resistant to change. Private privileges and profits make the planned constructions resistant to change; layers upon layers of accommodations and bargains that go by the name of tradition make the unplanned ones unyielding. But they are human constructions, and humans can destruct them or reconstruct them. The catastrophes send us warning signals. This book has attempted to decode these signals: abandon this, it is beyond your capabilities; redesign this, regardless of short-run costs; regulate this, regardless of the imperfections of regulation. But like the operators of TMI (three-mile island) who could not conceive of the worst -- and thus could not see the disasters facting them -- we have misread these signals too often, reinterpreting them to fit our preconceptions. Better training alone will not solve the problem, or promise that it won't happen again. Worse yet, we may accept the preconception that military superiority and private profits are worth the risks. This book's decoding asserts that the problems are not with individual motives, individual errors, or even political idologies. The signals come from systems, technological, and economic. They are systems that elites have constructed, and thus can be changed or abandoned. --Normal Accidents, Charles Perrow, 1999

---

[Jump to a Top Ten item: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10]

#8: The opaqueness of the infrastructure to empirical analysis has generated many problematic responses from rigidly circumscribed communities earnestly trying to get their jobs done.

1. To its credit, the IETF acknowledged and endeavored to solve the technical limitations of the current IPv4 protocol, primarily the insufficient number of addresses and the inherent scalability limitations of the routing architecture. To its chagrin, the IETF learned that neither the philospher king nor rough consensus-based approach would yield an architecture that made progress on both problems at the same time. So the IETF punted on the routing problems since they seemed further away, and focused on building a new network architecture that had a larger number of addresses, and some other stuff most people don't usually mention. But because today's addressing and routing architectures are fundamentally related, a larger number of addresses actually exacerbates the routing problem, getting us closer to the wall that seemed further away. In the meantime, the current IPv4 routing table is already splintering into smaller pieces as network operators engineer finer-grained control over traffic patterns. So, while IPv6 exists as a set of technologies, many experts are grim about its future, since it doesn't solve the fundamental routing scalability problem.
2. Most network operators, especially for-profit ones, cannot justify the investment to deploy IPv6 when their customers are not asking for it, and their customers won't ask for it until they can no longer get IPv4 addresses. Large network operators continue to remind IETF engineers that they didn't solve the problem the network operators really need solved. Operators do realize they are all in this together, but they aren't institutionally structured to think longer than five years out. They also lack the capital, legal framework, and incentive to develop an alternative replacement, even in partnership with their suppliers. (The last time we upgraded the network architecture the network was under the control of not only the U.S. government but the U.S. military. And it still took a couple of rounds of threats to cut off funding to attached sites who did not upgrade!) Instead, operators are busy experimenting with business models to try to figure out how to make a profit on IP transit, e.g.,fancy QOS services that customers aren't asking for, metered pricing (known to have its own problems), or giving up and getting rid of the part of the company that moves IP traffic around. They have also recently experimented with reforming their industry trade meetings to be more useful given that they aren't authorized to share any significant information about their own networks. In the meantime, if they have one, they heavly subsidize from the magnificently profitable wireless side of the company while they build the case for more deregulation.
3. Thinking about the health of the Internet ten years out or longer should theoretically happen within the stewardship missions of ICANN and the ICANN-rooted address registries, who lease Internet address space based on demonstrated need. The ICANN and registry communities recognize the limitations of IPv6, and by now also the limitations of the IETF. IETF experts are similarly astute about the problems with ICANN. And of course both communities are aware of the pressure on the current address space. Since IPv6 is the only existing solution, they both promote IPv6 deployment, although they lack reliable methods to measure IPv6 uptake without data from operators. So, this year they are finally re-discussing a backup plan: privatizing IPv4 address markets, in case they run out of IPv4 addresses before IPv6 gains traction. There is little background research on the implications of private ownership of addresses, but what exists is not auspicious. Furthermore, the possibility that a legitimate market for IPv4 address may emerge will itself impede the uptake of IPv6, so the bottom-up registries are inherently conflicted regarding the problem they're trying to solve.
4. Meanwhile, over in the media policy, reform, passionate activist, and well-intentioned legal scholar corner of cyberspace, it is as if Eli Noam's warning about the imminent death of common carriage were not published fourteen years ago. Despite the lack of any proposed operationally enforceable definition of network neutrality, the conversation thrives -- an understandable post-traumatic reaction to the recent jettison of at least eight centuries of legal doctrine from our primary communications fabric. Even the FCC is looking for ideas (strangely, they're explicitly not interested in data, despite clear indications that the free market evolution of IP economics is the root cause of the mess.) When the dizziness subsides, we will have to acknowledge that the carriers are right: it would be a disaster if the government told carriers how to manage congestion on their networks, which is why the endgame must be -- as it has always been with essential facilities and common carriage -- that carriers do not have financial interest in the content of what they're carrying. But that idea -- although it is the same type of structural regulation that made the Internet possible -- offends any capitalist sense of profit margins.
5. Academic Internet researchers also operate in a funding environment that does not promote tackling 10-year problems, nor are they equipped to navigate the conflict of interests between the university and the providers of network data. Providers either legally cannot or are reluctant to share data without restrictions on what can be published about their network, and universities have rules limiting such restrictions. And so federal agencies funding research continue to spend millions of R&D dollars per year developing lots of technology, even legal technology to promote data retention and sharing, but the agencies and the taxpayers they represent get little in return. A related problem is that the lack of experience with data sharing in an admittedly quite young field of science means that there is no established code-of-conduct for protecting user privacy and engaging with Institutional Review Boards to navigate ethical issues in Internet measurement research. Worse yet, conservative interpretations of the current relevant statutes conclude that most network measurement research is currently approximately illegal, but there is no consensus on what kind of legislative changes are needed, if any. The stunted legal process prevents sharing of data sets that could help solve immediate problems, but the collateral damage is that it prevents informed discussion of what even needs to be known on the net, and who needs to know it. Do we want to know how much peer-to-peer traffic is transiting backbone links? How much encrypted traffic? How much copyrighted traffic? Right now there is insufficient access to data to any of these questions. And answering them will come at a cost to the social contract of privacy. The conversation over how to make these tradeoffs has barely begun. For one, the academic community is too busy fighting lawsuits, the greatest incentive yet for universities to not retain data on network usage.So, while academic researchers do generate quite a bit of intellectually meritorious work, they are forced to choose scientific problems based on what data they can manage to scrape together (bottom-up) rather than picking the most important problems to study and getting the data needed to rigorously study them. Recently, a group of well-respected academics have become sufficiently desperate at their inability to study, modify, and share aspects of the Internet, that they've proposed building their own sandbox to develop and test innovative network technologies. It's like network neutrality at the research layer, an apparently irresistible attempt to recover some objectivity in the field, but in both cases symptomatic of the need for deeper inquiry .
6. The (predominantly libertarian) engineers in the router trenches have self-organized into squadrons of individual engineers and analysts: skilled, bright, principled people who until recenty mostly believed that if they worked hard enough, they could clean up the gutters of cyberspace without government intervention. Even these groups are now finally acknowledging that without better support for protected data-sharing, partnerships with government, and more educated law construction and enforcement, even their best efforts plus the market cannot fix the security problems. And although no one currently has positive expectations about the government doing any better anytime soon, neither are we in a position to claim the current lack of governance is working.
7. For the U.S. regulatory agency still reeling from the damage wrought by the 1996 (U.S. Telecom) act and its lifetime employment for lawyers, the opaqueness of the U.S. infrastructure, even to them, keeps them in the difficult position of trying to set policy in the dark. (Ironically, the FCC is the agency who should lead solutions to this problem, but as mentioned, their behavior suggests they want as little data as possible, since they have already made up their mind about how to (not) regulate the Internet.)
8. Innovative software developers move away from more oppressive legal frameworks, the net effect of which is to deprive the country of associated tax revenue and innovative climate.
9. Last but most important, the users, the youngest and most progressive of which are embracing activity that is arguably criminal under current legal frameworks. Although it is well-established that supporting and enforcing these legal frameworks (a tax-funded activity whose costs are unknown) does great economic damage while sacrificing privacy and freedom (not the best trade citizens have made), Hollywood insists (based on no verified data, natch), that on the contrary, it's the sharing of zero marginal cost goods that is causing the economic damage. While some governments admit they have no interest in tracking kids sharing music, for-profit entities now forced to partner with content providers for economic reasons (since as we know by now, you can't maximize profit just moving bits around) will find the temptation irresistible.

All these communities have tremendous insights into pieces of the problem, all are filled with earnest people trying to do their job, constrained by their institutional context. But no one has oversight for coordination or even articulation of the global picture. While the best available data makes it obvious that legal repair and renewal is crucial to democracy -- communications technology being no exception -- we are currently pursuing enlightened policy in the dark. Which begs the question: what is the most important ingredient to enlightened policy?

Such is the irresistible nature of truth that all it asks, and all it wants, is the liberty of appearing. Thomas Paine (1737 - 1809)

[Jump to a Top Ten item: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10]

#7: The traditional mode of getting data from public infrastructures to inform policymaking -- regulating its collection -- is a quixotic path, since the government regulatory agencies have as much reason to be reluctant as providers regarding disclosure of how the Internet is engineered, used, and financed.

For every other critical infrastructure in society we have devoted a government agency to its stewardship. The Internet was designed for a cooperative rather than competitive policy architecture, so its designers did not consider regulatory aspects. But as a communications infrastructure serving the public, most regulatory aspects of Internet fall under the jurisdiction of the agency who regulates the tubes it typically runs atop: in the United States that means the FCC. Unfortunately, the FCC is not completely up to speed on the Internet, and does not even approve of how it is measuring broadband penetration. The FCC has no empirical basis in fact nor apparent authority in a conversation about traffic, structure, pricing, or vulnerabilities on the network since it has no access to data from Internet infrastructure beyond what providers volunteer to provide. And yet little data is needed to reveal that the Internet's underlying network architecture, implementation, and usage is fundamentally inconsistent with almost every aspect of our current communications and media policy architecture. The Internet sheds deep skepticism on current legal frameworks for copyright, wiretapping, and privacy, as well as transforms or destroys dozens of industries that hold great economic and political power today.

The national security components of Internet regulation, from wiretapping to disaster recovery to unstable leadership lamenting its budgetary and policy handicaps, inspire concern than hope. That over 1% of observed web pages are modified in flight without our knowledge is no source of comfort either.

Hence it should be no surprise if solutions to measurement, like other persistent problems of the Internet, require engaging deeply with economics, ownership and trust issues. Alas, Internet economics research is one of the few fields worse off than Internet traffic or topology research with regard to the ability to validate any models or assumptions. (If you think tcpdump and traceroute are replete with measurement error, you should try analyzing the economics of network infrastructure companies. And if you think packet header and internal topology data is hard to get, you should try to get financial numbers from the same companies broken out by service offered so you could see how the economics are actually evolving.)

Unfortunately (again) understanding the economics of the system is not where spare private or public sector capital is going. In the 1990's the telecoms spent their capital suing each other and the government over laws so vaguely written as to defy consistent interpretation, much less measurable enforcement, across any two constituencies in the ecosystem. This decade we are spending our capital suing the telecoms for not suing the government after 9/11 when the government asked them to break laws that are just as outdated as the copyright laws. Thomas Jefferson would no doubt recommend rewriting all of it from scratch. Unfortunately the timing is bleak: these developments are occurring at a time when sustaining Internet growth (which, no, we still do not have good ways to measure..) will require extraordinary investment of capital, as well as realignment of incentives to promote cooperation among competitive players. Where does that capital and incentive to cooperate come from?

[Jump to a Top Ten item: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10]

#6: While the looming problems of the Internet indicate the need for a closer objective look, a growing number of segments of society have network measurement access to, and use, private network information on individuals for purposes we might not approve of if we knew how the data was being used.

To the extent that we are investing public or private sector dollars in trying to measure the Internet, they are not in pursuit of answers to questions related to the overall network infrastructure's health, system efficiency or end-to-end performance, or any of the questions that engineers would recommend knowing about a communications system. The measurements happening today are either for national security or business purposes, which both have an incentive to maximize the amount of personal information they extract from the data. No one is investing in technology to learn about networks while minimizing the amount of privacy compromised in the process.

This inherent information asymmetry of the industry is at the root of our inability to verify claims regarding either security or bandwidth crises justifying controversial business practices that threaten an admittedly fuzzy, but increasingly popular concept of Internet access rights. Although the little data that researchers can scrape together, most of it from outside the U.S., do not support the "p2p is causing a bandwidth problem" claim, the press releases we see as a popular substitute for real data in the U.S. do support the claim that the current Internet transit business model is broken.

Whether the growth in traffic is due to http transport of user-generated video, or radically distributed peer-to-peer file sharing (also often video), there is strong evidence from network providers themselves that the majority of bytes on the network are people moving files from machine to machine, often the same files moving from a few sources to many users. Unfortunately, this evidence implies that the current network and policy architectures are astonishingly inefficient, and that clean slate Internet researchers should be thinking about how to create truly scalable interdomain routing and policy architectures that are content-centric, leverage our best understanding of the structure of complex networks, and still manage to respect privacy. No easy trick, especially with no viable deployment path for such a new architecture, at least in the U.S. where we have jettisoned the policy framework that allowed innovations like the Internet.

It should be no surprise if the status quo is unsustainable, since we are using the network quite differently from how it was intended. But if a new network architecture is needed, that's a discussion that needs to include some validated empirical analysis of what we have already built. So long as the network infrastructure companies are so counterincented to share data, we will continue having to make trillion-dollar communication and technology policy decisions in the dark.