The Internet is a miracle

Taking a moment to appreciate everyday magic we take for granted

This post is a departure from my norm. It started as an aside in an upcoming “Progress” article I’m working on, and when I realized it was too much of a tangent, I thought I’d turn it into a note. When I finished writing, it was too long for a note, but I liked it enough that I decided I’d share it as a post. So here we are!

Some of you may know, I’m a former urban planner, and long-time “urbanism” advocate, but my day job is now in Software Engineering, specializing in web applications.

Today, in the spirit of Thanksgiving, I want to take a moment to appreciate what a marvel the Internet is. Yes, that Internet, the one you’re using to read this post. I know, old news, we’re used to it. At this point, we all take it for granted. But we shouldn’t!

Picture this. Somebody sends you a link to a viral tweet. You click on it, and it loads on your phone. You chuckle. But do you ever stop to think about what it takes to make that happen?

To help you appreciate this magical technology we all use every day, I’m going to give you a high-level walkthrough of the civilization-level effort required to load a tweet. Here’s how it works:

First, the app on your phone creates what’s called an HTTP Request. HTTP stands for “Hypertext Transfer Protocol.” It’s a way of formatting messages for computers to send back and forth that was originally designed for serving web pages (hence the “hypertext”), but has been extended to cover the majority of traffic on the Internet.

An HTTP Request is plain text. To load some page or data from the web, you’d normally send a GET request, which looks like this:

GET /DennysDiner/status/1989813917359509530 HTTP/1.1
Host: x.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0 Safari/605.1.15
Accept: text/html

That request specifies the content your phone wants to load. There’s another computer, somewhere out in the world (probably in a datacenter in Northern Virginia), that can respond to that request and send back the content. But neither you nor your phone know where that computer is. So how do you send it your message?

Finding and connecting to that other computer is the problem the Internet solves.

The HTTP Request specifies a “host,” which is the human-readable domain name used by that other computer. To send that computer a message, it needs the Internet Protocol (IP) Address for that computer, so it performs a Domain Name Service (DNS) Lookup, which involves recursively searching a series of databases until it finds a match.

First, the phone checks its own database. If you’ve loaded something from “x.com” recently, the phone has already looked up the IP Address and can use it again. If not, it sends a message to ask the nearest cell tower, which in turn asks the cellular provider’s network, which in turn asks the other internet providers it’s connected to, and so on, until one of the computers in that chain finds the address and sends it back down the chain. Finally, your phone will learn that the IP Address for x.com is 172.66.0.227.

Note that this involves two other technological miracles that we’re going to skip right past:

• Your phone is using a cellular network, which means it can’t just send messages over a wire like the internet was designed for, it has to broadcast radio signals to, and receive radio signals from, cellular towers that are wired to the internet. That uses an entirely different, but equally amazing, stack of networking technology.

• Actually, “x.com” has many IP addresses, and deciding which one your phone will get is handled by a service called Cloudflare; another technological marvel that solves a whole category of problems related to serving more internet traffic than any one computer could ever handle.

Once your phone has the IP address, it can send packets — tiny messages — to that address. So the phone sends packets through the cellular network to the cell tower.

At the cell tower, packets from your phone are read by a special kind of computer called a router. Each router is like a post office: it reads each packet to check the address it’s trying to reach, then has to decide which computer to send it to next. The router is connected to other routers, which are in turn connected to other routers, and so on. The router’s job is to forward packets it receives to whichever router is the next closest to the packet’s destination.

But how can it know which one that is? To figure it out, the router uses a routing protocol, which works roughly like this:

The router asks each router it’s connected to if that router can reach 172.66.0.227. Each of those routers repeats this step, forwarding the request to all the other routers it is connected to, and so on. This search ripples out through the internet until some router answers “yes,” it can connect to the computer with that address. Then all the devices in the chain each report back that they found the destination, and how many “hops” it took them to get there. The first router then sends the packets to whichever connected device says it needed the fewest “hops” to get there. That continues for hop after hop, until dozens of routers have passed your message all the way from your computer to the computer that’s listening at the address 172.66.0.227.

Note that those packets are traveling through physical cables that are stretched across every continent, and in between continents via vast submarine cables that span the world’s oceans!

Now that the routers have found the path from your phone to the computer with the address 172.66.0.226, the two can send and receive packets. But, things can happen, and the packets can get mixed up or arrive out of order. To deal with that, the two computers use Transmission Control Protocol (TCP), which (basically) lets the two computers number the packets so they can be put in the right order if they got mixed up in transmission, or re-send any packets that got lost.

Now data is flowing across the network, but there’s one more problem. All of this data is plain text, and all the computers in between your phone and the other computer can read all the messages you’re sending back and forth. Maybe that’s fine for some content, but it’s definitely not fine if you’re using a banking app! To keep the data private, the computers use Transport Layer Security (TLS) to encrypt the connection. This is possible thanks to public key cryptography, which is now the third amazing technology that I’ll skip over, except to say that it guarantees the data flowing between the two computers can only be read by them, and not the many intermediate devices that pass that data along.

Finally, at the end of all of this, your phone can load that viral tweet.

All that accumulated progress, and I haven’t even talked about the operating systems, the computer hardware (and its vast industrial supply chain), the telecommunications infrastructure, and the electrical grid itself, all of which are necessary for you to read a tweet. So once in a while, when you load something on the web, think for a moment about how great civilization is, that we can produce systems so powerful and reliable that we take them for granted.

Happy Thanksgiving, everyone!

Thanks to Sam Enright for his feedback. And if you’d like an even deeper dive on how the Internet works, try the Mozilla Developer Network’s guide.