curl is an exceptionally useful program. As described on the project homepage (https://curl.se/), curl is a tool to transfer data from or to a server, using one of the supported protocols. curl can be used to send & receive data with the following protocols:
- DICT
- FILE
- FTP
- FTPS
- GOPHER
- HTTP
- HTTPS
- IMAP
- IMAPS
- LDAP
- LDAPS
- POP3
- POP3S
- RTMP
- RTSP
- SCP
- SFTP
- SMB
- SMBS
- SMTP
- SMTPS
- TELNET
- TFTP
To better explain curl, I will demo curl on Ubuntu 14.04. First, I will execute curl http://gitlab.com
The command executes very quickly. However, a lot is actually being performed in the background.
curl http://gitlab.com
When the user executes curl http://gitlab.com, a request is sent from the application (i.e. curl) to the kernel.
The kernel acts as a middle man between the system’s software applications and hardware. curl needs to talk to some of the hardware components, including the CPU, memory, and network adapter. Given that curl is a network application, the kernel definitely needs to talk with the network adapter. When curl http://gitlab.com is executed, the user is telling curl to send some data over HTTP, in hopes of a response.
In order to visualize the HTTP data being sent to http://gitlab.com, I will use WireShark to sniff the outgoing packets. Below is the packet capture performed while executing curl http://gitlab.com:
My computer is currently using Google’s public DNS server (8.8.4.4). Given that gitlab.com is out on the Internet, my computer has to send a DNS request to Google so the domain name can be translated to an IP address. Gitlab.com appears to be at 52.167.219.168. Now that my computer knows the IP address of gitlab.com, HTTP requests can be sent. The curl HTTP requests go from my computer, to my ISP gateway, and out to 52.167.219.168. The processes appear to have completed and data was received.
Let’s see what WireShark collected from the HTTP request:
The text highlighted in red was the request my computer sent to gitlab.com’s server. The text highlighted in blue is what was returned from gitlab.com’s server. When the request was sent to gitlab.com’s server, gitlab.com returned a 301 status. The HTTP 301 status code means Moved Permanently. This code is usually thrown when an user accesses a site and the web server redirects them to another. To prove this holds true, let’s open a web browser and go to http://gitlab.com.
When launching the HTTP request from my web browser, the server redirected me to this:
The URL http://gitlab.com takes you to the home page of gitlab.com:
Hard to believe all of that happens just by executing a small command! The preceding was the whole curl process, from the kernel to over the network, and completed on gitlab.com’s server.
(Originally posted on June 8th, 2017. Updated on December 29th, 2020)