The thunder of distant Net storms

Marcelo O. Magnasco Center for Studies in Physics and Biology The Rockefeller University 1230 York Avenue, New York NY10021, U.S.A.

December 1, 1999

Abstract Computers and routers on the Internet send each other error messages (called ICMP datagrams) to signal conditions such as network congestion or blackouts. While these datagrams are very rare, less than 0.001% of total traffic, they hold very important global information about problems and congestions elsewhere in the Net. A measurement of the flow of such error messages in our local cluster shows a very pathological distribution of inter-message times: P (Δt) ≈ 1/Δt. This scaling extends for about seven decades, and is only punctuated by extraneously periodic signals from automatons. More than a half of these error messages were themselves generated erroneously.

The decentralized and anarchich structure of the Internet has, for a long time, invited people to perform all sorts of measurements. Indeed, it has taxed and even challenged our ability to make such measurements: for instance, it’s been several years since we last knew, to any accuracy, how large the Internet actually is [?, ?]. Many types of measurements have shown “anomalous” or “pathologic” statistics, meaning that the distribution of some quantity has power-law (a.k.a. Pareto or “heavy”) tails . A power-law tail implies that the probability of an outlier diminishes very slowly with the size of the outlier, and hence measurements show strong irregularities. While anomalous from a classical statistics viewpoint, this is a very usual phenomenon in the natural sciences, and has been extensively studied [?, ?]; in particular, one of the earliest examples concerns noise in communication channels [?]. As applied to the Internet, local traffic measurements at the datagram level [?, ?, ?, ?, ?], session data [?], webserver workloads [?, ?], USENET thread length [?] and Web surfing patterns [?] all show these tails. There is ample evidence that these statistical pathologies do not come from intrinsic instabilities of the Net as a communications system, but rather stem from the way in which people use the Net. Power-law distributions are often regarded as prima-facie evidence of some self-organizational process, but they can also simply be a reflection of other underlying power laws. The sizes of all files on a computer follow a power-law distribution, from small files a few characters in size, to large datasets (e.g., videos) hundreds of megabytes in size. It is thus natural that, for instance, the sizes of documents retrieved from webservers reflect this underlying breadth [?], and it’s also been shown that individual surfers follow anomalous Levy-flight-like surfing patterns [?]. A useful and clarifying distinction was made in [?]: the initiations of “sessions” (such as telnet or ftp) follow perfectly normal Poissonian patterns, while the individual transactions within a session (the ftpdata connections, or the TCP traffic within a given telnet session) fluctuate over many scales.

However, all such measurements of local “traffic” have built-in limitations as to how much they can fluctuate. Indeed examination of the data shows a fairly complicated picture: the bulk of the traffic is not “self-similar” or pathological; only some tails show power-law scaling over a limited range. See Figure 1. Even a moderate amount of scaling behaviour can certainly create a lot of trouble for engineers (who have to design equipment to handle such contingencies); but it is not pervasive enough, nor is it defined over a large enough range, to qualify as “self-similar” behaviour in the sense in which it is usually used in the natural sciences. We could say that we are trying to make a measurement of precipitation during a storm: it will probably fluctuate and be gutsy, but in its bulk be statistically regular.

We will do something different here: we will not focus on what we can mea-