As described in Section 3, the Internet can be viewed as an interconnection of national and regional networks, end-users and organizations, and interexchange points. The Internet is a very dynamic entity that is constantly evolving and growing. Therefore, it is impossible to identify all of the components of today's Internet. For this report, the Internet is analyzed to identify key components used to transmit network traffic across the Internet. To achieve this purpose, a software tool, referred to as the IAT, was used to automatically trace the routes used to send traffic between two hosts on the Internet. The tool collects the set of routers an IP packet traverses on its path from one host to another. The analysis of these routes will identify traffic trends and key components in the Internet infrastructure.
This section provides an in-depth description of the IAT and analysis results. Section 4.1 details the functionality of the IAT. Section 4.2 details the implementation of the tool, including the set of hosts that was analyzed. Section 4.3 presents the analysis methodology and the results of the analysis.
4. 1 INTERNET ANALYSIS TOOL FUNCTIONALITY
The purpose of the IAT is to collect the routes traveled by IP packets from one host to another. Because it is impossible to collect and analyze routes between every host on the Internet, a subset was chosen to provide an accurate sample of U.S. Internet traffic. Section 4.2 details the sites chosen for this analysis.
The IAT utilizes a UNIX utility, traceroute, to record the different routers a packet traverses once it is sent from the originating host to the destination host. The traceroute application is available with all UNIX and UNIX-variation operating systems. traceroute uses the Time To Live (TTL) field in the IP packet header to determine the routers in a particular path. The purpose of the TTL field is to ensure that packets do not stay on the Internet for an infinite amount of time (e.g., as a result of a routing loop). Each router that receives an IP packet is required to decrement the TTL field in the IP header by the number of seconds the router holds onto the datagram. Because most routers process a datagram in less than one second, the TTL field effectively becomes a hop counter that is decremented by one by each router.
IP packets are usually transmitted with a TTL of 60 by the originating host. When a router has an IP datagram with a TTL of one, the router decrements the TTL to zero, discards the packet, and returns an error message to the originating host. This error message is an Internet Control Message Protocol (ICMP) packet that identifies the router that sent the error message and indicates that the time has been exceeded on the datagram.
The basic operation of the IAT is to send out traceroute IP datagrams beginning with a TTL of one, then a TTL of two, and so on, until the entire route between two hosts is determined. The router receiving the first IP datagram with a TTL of one will decrement the TTL and return an ICMP message to the originating host. This identifies the first router in the path. The IAT will then send out a second traceroute IP datagram with a TTL of two. The first router decrements the TTL to one, and sends the datagram to the next router in the path. The second router will decrement the TTL to zero and return the ICMP message. This continues until enough datagrams have been sent to have one of them reach the destination host. The destination will not discard the traceroute IP datagram, even though it will have a TTL of one because the datagram is addressed to that host.
For the IAT to determine that a datagram has reached its destination (because it has not received the final ICMP message), the IAT sends UDP datagrams to the destination host using a very high destination port number. The destination host will not respond to incoming packets on this port number; thus, the destination host will send back an ICMP "port unreachable" error to the IAT. The IAT differentiates between the time exceeded and port unreachable errors to determine when the route has been fully traced.
The output from an IAT execution is the set of routers in the path between two hosts. For each router, three datagrams are sent, and the round trip time from the originating host and the router is collected. Exhibit 4-1 depicts the sample output from a source host to the destination, www.disa.mil.
Traceroute to www.disa.mil 1) Cisco-AGS.dcmetro.bah.com (156.80.1.1) 2 ms 2 ms 2 ms 2) fr.herndon.va.psi.net (38.2.104.1) 128 ms 227 ms 47 ms 3) 38.1.2.19 (38.1.2.19) 45 ms 77 ms 138 ms 4) mae-east.ddn.mil (192.41.177.130) 68 ms 54 ms 41 ms 5) 137.209.1.2 (137.209.1.2) 176 ms 168 ms 204 ms 6) 198.26.127.10 (198.26.127.10) 179 ms 132 ms 114 ms 7) 164.117.2.13 (164.117.2.13) 134 ms 127 ms 134 ms 8) 164.117.1.1 (164.117.1.1) 143 ms 135 ms 125 ms 9) www.disa.mil (164.117.147.116) 135 ms 147 ms 176 ms |
4. 2
INTERNET ANALYSIS TOOL IMPLEMENTATION
This section details the implementation of the IAT described in Section 4.1. For this analysis, two sites were chosen as source sites:
* Booz-Allen & Hamilton, McLean, Virginia, on the PSINet network
* Proxima, Inc., McLean, Virginia, on the MCI Network.
The tool collected routes from each of these two sites to 105 other sites located across the United States. The Web sites chosen for this analysis included the following:
* 23 NCS Member Organizations Web sites
* 50 State Web sites
* Major university Web sites
* Popular commercial Web sites.
Appendix A provides the entire list of Web sites used in this analysis. Exhibit 4-2 also shows the geographic locations of these sites. The IAT, which collects the routes from the two source locations to all 105 sites, is executed six times daily, every four hours beginning at midnight. This results in a sample of Internet traffic throughout the day. The output from this tool is formatted and loaded into an Oracle database where the analysis on the collection of routes is performed. Section 4.3 presents the analysis methodology followed by the IAT study.
4. 3 INTERNET ANALYSIS RESULTS
The data collected using the IAT represents a general picture of Internet connectivity. The destination Web sites used in the analysis were selected to provide both a United States and NCS specific view of the Internet's topology.
4. 3 .1 Internet Analysis Methodology
An in-depth analysis of the physical topology of the Internet would be an incredibly complex and difficult task. Because of the number of national backbones and regional distribution networks spanning multiple carriers, the Internet's topology is an amalgamation of CLEC, ILEC, and IEC networks. Determining the entire physical topology of the Internet may well be impossible without the cooperation of these PN carriers.
An analysis methodology was developed to provide the most complete and valuable view of the Internet and its topology. The methodology defines the steps used to evaluate the data obtained from the IAT. A description of the methodology is presented below.
* Identify Scope of Analysis. Although the Internet is too large and complex to be handled in its entirety, the scope of the analysis was selected to provide a representative view of the Internet. The IAT is most useful in analyzing single, specific routes, not large network topologies. Given enough representative routes, the collective results of the IAT can provide a view of portions of the larger Internet. By collecting data at various times of day from multiple routes, the IAT provides a representative set of data. The originating and destination sites selected for this analysis provide a distribution of sites across the United States. The inclusion of the NCS Member Organizations provides a capability to capture and analyze data specifically for the NCS community.
* Identify Pertinent Data. The data used in the analysis must provide a complete and accurate picture of how IP packet traffic will be routed over the Internet. Variables such as the distance traveled, number of networks traversed, and the congestion of the network will affect how packets traverse the network. The IAT provides a host of data that is used to analyze our representative Internet routes. The data used in this analysis includes the following:
– Origin to destination route
– Physical distance of route in air-miles
– Time of day
– Round trip time
– Number of hops in route
– Routers in route
– Networks in route.
* Identify Valuable Results. The purpose of the analysis is to identify the discriminating variables that affect the Internet's performance. Using the available data (i.e., round trip time and number of hops in route), the analysis should indicate differences in performance based on the following variables:
– Critical nodes
– Physical distance between hosts
– Time of day congestion
– Number of networks traversed
– Relative size of networks traversed (NSP versus RSP).
These results will provide input to the analysis of the vulnerabilities of the Internet. They may also identify how the OMNCS and NCS Member Organizations can improve Internet reliability by choosing certain ISPs, mirroring important Web sites, or performing downloads in off-peak hours.
Exhibit 4-3 illustrates the Internet analysis methodology.
Exhibit 4-3
Internet Analysis Methodology
[click here to view exhibit 4-3]
4. 3 .2
Internet Analysis Results
The IAT analysis focuses on identifying the path that is critical for transmitting data across the Internet's regional and national backbone networks. The data provided by the IAT was analyzed to trace the paths through the Internet and to identify how Internet data traffic is affected by daily traffic surges and congestion and network outages. This analysis is intended to provide an estimate of the performance characteristics of a portion of the U.S.-based Internet. However, the results presented here cannot be assumed to represent the entire Internet, or even the entire U.S.-based network. This is because of the limited scope of the data, and the sheer size of the Internet in terms of routers and hosts. More thorough analyses of the entire Internet are planned as a follow-up to this initial analysis. We chose to use two source hosts for this analysis, one based on Booz•Allen & Hamilton's network and the other on Proxima, Inc.'s network. Booz•Allen and Proxima, Inc. receive Internet service from two of the six NSPs, PSINet and MCI, respectively. Therefore, this analysis may primarily represent the characteristics of these two networks.
The data provided by the IAT traces is the basis of a statistical analysis of the number of hops and round trip time for the 210 source and destination pairs (2 sources and 105 destinations). Traces were given a status of either "successful" or "unsuccessful." A successful trace was one in which the IAT packets generated reached the destination router address and an unsuccessful trace was one in which they did not. Exhibit 4-4 shows the number of successful traces per source and the percentage of the total traces performed.
Source | Total Traces | Successful Traces | Unsuccessful Traces |
Booz-Allen | 5134 | 4468 (87 %) | 666 (13 %) |
Proxima, Inc. | 9128 | 8098 (88.7 %) | 1030 (11.3 %) |
A small percentage of the traces for both sources was determined unsuccessful. An unsuccessful trace could typically be attributed to one of the following reasons:
* The destination name server entry could not be resolved and therefore the trace never began
* An initial router of the ISP could not be reached
* A router or gateway in the path of the trace was unreachable
* The destination server was unreachable, most likely due to it being shut down
* The host's network might use code that is incompatible with the IAT testing protocol. That might have resulted in a router not returning the ICMP messages required for the operation of the IAT.
Exhibit 4-5 illustrates an approximate categorization of reasons why traces were unsuccessful. The percentages of those due to an unreachable path router, an unreachable destination server, or incompatible network code were combined. A hop-by-hop analysis of all unsuccessful traces, comprising nearly 45,000 hops, would be required to determine the component percentages.
Booz-Allen | Proxima |
Unresolved host name: | 2.6 % | 0 % |
ISP unreachable: | 0.2 % | 0.8 % |
Router or destination machine unavailable: | 10.2 % | 10.5 % |
Total Unsuccessful: | 13.0 % | 11.3 % |
The results described in the remainder of this analysis are solely based on successful traces.
4. 3 .2 .1 Traffic Congestion
Internet traffic encounters congestion due to surges in its use in daylight hours. Traffic surges occur during working hours, and most notably between noon and 6:00 p.m. Weekend traffic should not be as susceptible to Internet congestion because of the reduced number of business users. Our analysis assumes the effects of congestion will become manifest in the response time for data traveling over the Internet.
The IAT collects the round trip time for a single datagram to travel to and from each of the destinations. For each destination, three datagrams are sent, and the total travel time is recorded for each. The average travel time versus time of day for these datagrams is shown in Exhibit 4-6. As expected, these results appear to coincide with traffic patterns for a typical east coast IXP, MFS's MAE-EAST. The additional traffic on the Internet results in a proportional increase in the delay time. Representative weekday and weekend data for MAE-EAST and MAE-WEST are shown in Exhibit 4-7 and Exhibit 4-8, respectively. The traffic increase between 12:00 noon and 4:00 p.m. shown in the MAE-EAST traffic profile is similar to that of our round trip time results. Note that traffic on MAE-EAST, located in Washington, DC, and MAE-WEST, located in San Jose, CA, are nearly identical for the time of day, based on eastern standard time (EST). Because of the large amount of traffic traveling between the east and west coasts, these two IXPs are interdependent. The traffic generated on the east coast between 12:00 noon and 4:00 p.m. eastern time affects the west coast traffic patterns between 9:00 a.m. and 1:00 p.m. Pacific time.
Exhibit 4-6
Average Round Trip Time Versus Time of Day
[click here to view exhibit 4-6]
Exhibit 4-7
Typical Traffic Patterns at MAE-EAST
[click here to view exhibit 4-7]
Source: MFS Datanet, Inc.
Source: MFS Datanet, Inc.
4. 3 .2 .2
Network Outages
Network outages are the most disruptive of the Internet's vulnerabilities. In the case of critical nodes, a network outage can preclude access or egress from the network (as in the case of an isolated regional or local network) or severely hamper the flow of traffic (as in the case of a NAP or IXP failure). Network outages will occur with much less frequency than network congestion, but they may result in a significant reduction of network capacity and availability depending on their severity.
We determined the number of hops in each successful IAT trace from source to destination. Exhibit 4-9 compares the average number of hops with the time of day for each source network. It is clear that the number of hops does not depend on the time of day. This indicates that the path taken from source to destination does not change frequently due to outages or routing around network congestion. This is because Internet routing tables are generally static. Routing tables are meant to change during a disruption in service and in the event of network congestion. Although some routing algorithms will route around link congestion, this analysis indicates this is uncommon, because the number of hops does not depend on the time of day, while congestion does. Creating large Internet routing tables requires expensive processing power. This process can result in more route "thrashing" than actual routing. In fact, routing tables will normally only be recreated when links become disrupted, or when a network administrator manually replaces the routing table.
We hypothesize that an outage in a critical network node, such as a national IXP, would greatly reduce, but not eliminate, the ability of the Internet to route traffic quickly nationwide.
4. 3 .2 .3 Critical Network Nodes
As explained in Section 3.2, the Internet relies primarily on the national IXPs to route and exchange traffic. Exhibit 3-4 shows the locations of the major IXPs across the United States. These high-speed LANs provide the majority of the routing among the backbone NSPs and the RSPs. Additionally, traffic is exchanged at private direct connects between ISPs' networks. Private direct connect exchange points of this kind are becoming more common due to congestion at the IXPs. ISPs are establishing private direct connects to avoid congestion problems and improve routing redundancy.
The IAT output provides the IP address of each of the routers traversed in the Internet traces. Using this data, we compiled lists of the most commonly visited routers for our two source networks. Exhibit 4-10 shows the distribution of the normalized frequency of use for the top 50 routers for both sources. The normalized frequency was obtained by dividing the number of hits on any router by the total number of hits recorded by the IAT for that source. This allows a direct comparison between the two sources. The first, second, and third router in each trace is considered to be specific to the source. These three routers show a very high frequency of use for our sources, and they are therefore critical to these sources, but do not fairly represent the remainder of the Internet. These three routers have been eliminated from the remainder of this analysis.
The network domain names provided by the IAT output identify the router's owner. Using these domain names, we identified the relative importance of ISP networks to the two sources. The normalized frequency of use for each network is shown in Exhibit 4-11. "Other" networks were those networks that did not individually represent a large portion of the total frequency of use or that were not identified by a domain name.
The critical network nodes had the highest frequency of use. Network nodes were considered critical if:
* They had a high frequency of use
* They were not too specific to the source routes, i.e., the top three routers
* They were not too specific to the destination routes.
All routers with normalized frequency greater than 0.004 were considered critical. Each of the sources shows a dependence on multiple critical network routers to trace a path to the destinations. Exhibits 4-12 and 4-13 show the critical ISP networks for the Booz•Allen and Proxima ISPs, respectively.
Some of the critical nodes for one source were also critical to the other source. These nodes become our most critical nodes, which we can then identify as critical to the Internet based on our study. Exhibit 4-14 shows the distribution of these critical nodes to ISP networks.
Exhibit 4-14
Shared Critical Nodes
[click here to view exhibit 4-14]
4. 3 .2 .4 Conclusions
Traces performed throughout the test period indicated high success rates averaging between 87 and 89 percent. Of the unsuccessful trace attempts, most were due to an unreachable node (i.e., a router or the destination server) in the path that was probably either shutdown or incompatible with the IAT software.
Internet use is highest during mid-to-late afternoon business hours. Based on the round trip time for packets to traverse the network, congestion peaks between the hours of 12:00 noon and 4:00 p.m. eastern time. However, the dependence of businesses on the Internet could not be determined, i.e., the analysis did not determine whether the Internet was used to conduct critical business communications and research, or simply for personal use.
This analysis indicated that the number of hops did not depend on the time of day or the day of the week. Generally, routing tables are rarely modified to route around network congestion. Unlike switched traffic, the routes of Internet connections were somewhat "predictable." Therefore, the predictability of Internet routing, along with an increasing dependency on this communications media, renders it vulnerable to targeted and intended network disruptions.
Routers appear to share a somewhat balanced traffic load within the backbone networks (excluding those routers closest to the two sources). As expected, a high number of router "visits" occurred in the initial hops of the traces. These initial routers are critical to the sources, however, they are not necessarily critical to the entire Internet. As the trace moved away from the source and into the backbone networks, the number of visits per router stabilized. Therefore, a single critical router could not be identified, however, it could be determined which networks were more heavily traversed. For this analysis, MCI's network was traversed most frequently and was therefore critical to the success of the traces.