The Deep Web

If you watch CSI: Cyber, you may be asking, as my mother does, what, exactly, is the deep web? Well, to answer yours and mom’s curiosity, the deep web is a part of the internet that is hidden, but in plain sight. As the show has often featured in its story lines, the deep web can be a place for nefarious activities, such as the selling of illegal drugs or black market weapons. While that type of activity leads to interesting TV shows, it is not the entire deep web, nor is it the reason the deep web exists in the first place.

In simplest terms, the deep web is a set of websites that are not searchable by using Google, Bing, or any other search engine. There are essentially two parts to the deep web: the hidden web and the dark web. The hidden web is mainly a set of websites that the owners do not want to make fully available to the general public, while the dark web is harder to find and is where those CSI characters look for illegal activity. In both cases, the website owners have purposefully used technology to block the sites from appearing in search results, either by using software or simply not setting up the site to show up in search results. More on this in a minute, since a little history is needed to understand the purpose of the deep web.

The internet was created in the 1960s as a means to ensure military bases could still communicate in case of invasion or attack during the Cold War. DARPA, the U.S. government agency that researches emerging technologies, worked with some big-name research universities to develop the technology that was used to create the first computer-to-computer network, ARPANET (named after the agency’s name at the time). The network went live in 1968 and was nationwide in 1970, with a goal of connecting key research and defense centers to share information and serve as that much-needed communications hub in a time of crisis. Of course, we all know the technology and backbone went in a completely different direction and by the 1990s, some of us early adopters were signing up for Compuserve email accounts. For more about the history of the creation of the internet, check out the book Where Wizards Stay Up Late.

Today’s deep web has its origins in the initial intended purpose of the internet. While the whole structure has become a very public place, there still needed to be someplace where the sites used for defense purposes or to share research could be secure without the average Google user stumbling upon them. Hence, the creation of the software and network protocols that keep the deep web hidden from the majority of computer users. But, that is not to say the average person cannot access or use the deep web. In reality, many hidden sites can be accessed with a regular computer using any internet connection, provided you have the address and any passwords needed to access the website. Most likely, the typical person will not be able to check out, even just for curiosity’s sake, the black market arms dealer websites. Yet, you can explore a bit on the deep web and put some of the technology to use for your own security.

Let’s start with a basic website that can get you a feel for the deep web right away: duckduckgo. This is a search engine designed to protect your privacy, a so-called proxy site that will take you to a website indirectly so you avoid cookies and other ways you can be tracked. We have all had the experience of looking something up on a search engine and then the very next time we go to a website with ads, all we see are ads that relate to that search or we get ads right in the search engine itself. While duckduckgo is not hidden or cloaked in the dark web, it also does not using tracking software so whatever you look for will not be retained or sold for marketing purposes. The site can be customized so you may elect to save information, but that also will not be shared with marketers.

If you want to stay totally hidden, then you have the option of using software, such as kind that runs the TOR network. TOR (the onion router) is basically a network on top of a network (get it, peel the layers), one that uses software to encrypt web addresses to disguise their real location and keep them out of search engines. TOR addresses do not end in a .com or any other typical domain type, but end in .onion. (No, not the humor website The Onion). An .onion address looks a bit like a bit.ly or tiny.url in reverse – instead of shortening a long address to a set of a few numbers or letters, the TOR network uses long strings of letters and numbers followed by the .onion “domain”. In order to view these websites, you will need the free TOR browser provided on the TOR project website. In addition to the browser, TOR provides more advanced software for running TOR sites, etc. While TOR was created with help from DARPA, it is now a non-profit and has become a tool for both legal and illegal activity. When you see a news story about a black market site like Silk Road, it is most likely using the TOR software to evade detection. Then again, TOR is also used by human rights protestors located in some not-so-democratic areas. TOR users have come up with a variety of ways to list TOR sites and some big websites, such as Facebook*, have created and published onion addresses for their users who wish to be more secure. Even a basic Google search on the surface web will reveal many of these resources.

Of course there are probably more deep web technologies in use that we do not even know about. At least now you can feel a bit smarter the next time you watch CSI: Cyber, Homeland or even wonder just how they do that when you see something on the news about a site like Silk Road or an uprising in another country.

*The Facebook TOR address is - https://facebookcorewwwi.onion/ and users of Facebook can read about the onion address here.

- Laura N.