*Gothic Gothic Gothic! All "Twilight," all the time!
http://www.guardian.co.uk/technology/2009/nov/26/dark-side-internet-freenet
(...)
"The darkweb"; "the deep web"; beneath "the surface web" – the metaphors alone make the internet feel suddenly more unfathomable and mysterious. Other terms circulate among those in the know: "darknet", "invisible web", "dark address space", "murky address space", "dirty address space". Not all these phrases mean the same thing. While a "darknet" is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of "the deep web", spooky as it sounds, consists of unremarkable consumer and research data that is beyond the reach of search engines. "Dark address space" often refers to internet addresses that, for purely technical reasons, have simply stopped working.
And yet, in a sense, they are all part of the same picture: beyond the confines of most people's online lives, there is a vast other internet out there, used by millions but largely ignored by the media and properly understood by only a few computer scientists. How was it created? What exactly happens in it? And does it represent the future of life online or the past?
Michael K Bergman, an American academic and entrepreneur, is one of the foremost authorities on this other internet. In the late 90s he undertook research to try to gauge its scale. "I remember saying to my staff, 'It's probably two or three times bigger than the regular web,"' he remembers. "But the vastness of the deep web . . . completely took my breath away. We kept turning over rocks and discovering things."
In 2001 he published a paper on the deep web that is still regularly cited today. "The deep web is currently 400 to 550 times larger than the commonly defined world wide web," he wrote. "The deep web is the fastest growing category of new information on the internet … The value of deep web content is immeasurable … internet searches are searching only 0.03% … of the [total web] pages available."
In the eight years since, use of the internet has been utterly transformed in many ways, but improvements in search technology by Google, Kosmix and others have only begun to plumb the deep web. "A hidden web [search] engine that's going to have everything – that's not quite practical," says Professor Juliana Freire of the University of Utah, who is leading a deep web search project called Deep Peep. "It's not actually feasible to index the whole deep web. There's just too much data."
But sheer scale is not the only problem. "When we've crawled [searched] several sites, we've gotten blocked," says Freire. "You can actually come up with ways that make it impossible for anyone [searching] to grab all your data." Sometimes the motivation is commercial – "people have spent a lot of time and money building, say, a database of used cars for sale, and don't want you to be able to copy their site"; and sometimes privacy is sought for other reasons. "There's a well-known crime syndicate called the Russian Business Network (RBN)," says Craig Labovitz, chief scientist at Arbor Networks, a leading online security firm, "and they're always jumping around the internet, grabbing bits of [disused] address space, sending out millions of spam emails from there, and then quickly disconnecting."
The RBN also rents temporary websites to other criminals for online identity theft, child pornography and releasing computer viruses. The internet has been infamous for such activities for decades; what has been less understood until recently was how the increasingly complex geography of the internet has aided them. "In 2000 dark and murky address space was a bit of a novelty," says Labovitz. "This is now an entrenched part of the daily life of the internet." Defunct online companies; technical errors and failures; disputes between internet service providers; abandoned addresses once used by the US military in the earliest days of the internet – all these have left the online landscape scattered with derelict or forgotten properties, perfect for illicit exploitation, sometimes for only a few seconds before they are returned to disuse. How easy is it to take over a dark address? "I don't think my mother could do it," says Labovitz. "But it just takes a PC and a connection. The internet has been largely built on trust." (...)
