How To Download Files In A Terminal Using The WGET Command

Table of Contents

Just when you thought that the terminal can’t get any niftier, here is another banger command that makes life easier. wget is a CLI utility that can download files from the web. wget can easily download files, websites, and even entire FTP directories. It supports HTTP, HTTPS, and FTP protocols. Unlike using a browser to download files, wget doesn’t need to be babysat, as it can work in the background retrieving files non-interactively. In today’s post, we’ll explore some of the useful features of wget and walk through a few examples to get started.

Installing wget

wget comes pre-installed on most distros* but if for some reason you don’t have it installed on your system you an use one of the following commands to get that taken care of:

Arch

				
					sudo pacman -S wget
				
			

Ubuntu

				
					sudo apt install wget
				
			

RHEL

				
					sudo yum install wget
				
			

wget Syntax

Personally I’ve used wget primarily in the context of installing CMSes on web servers so let’s try doing that! Below we will download the latest version of WordPress in the current present working directory.

				
					wget https://wordpress.org/latest.zip
				
			

Pretty straight forward right? Simply put wget in front of the URL of the file you want to download. Much like other commands we’ve come to know and love, you can also perform multiple download operations at the same time, by seperating the targets with a space:

				
					wget https://URL1 https://URL2 #and so on
				
			
Of course that is just the tip of the wget iceberg, as it is a very robust utility.

wget options

Continuing Downloads

Much like a dedicated browser download manager, you can continue downloads if they are interrupted for any reason. Pass the -c or --continue switch to wget and it will attempt to re-download the file where it left off.

				
					wget -c https://wordpress.org/latest.zip 
				
			

I’ve found that it doesn’t really matter where the option is placed, either before or after the target URL responds correctly.

As you can see wget continued at the point where the download was interrupted and resumed from that point. Note that the progress bar visually changed to indicate this.

Mirroring a site

Let’s try something more exotic. wget can do way more than download single files, it can also follow links in HTML & CSS recursively. That means with the right parameters you can mirror an entire site to your hard drive for offline viewing. Now before we try this out, let’s create a new directory. This is done because when we point wget at a URL, oftentimes we end up downloading a lot of files, so to keep things tidy we’ll just work out of a directory called sandbox:

				
					mkdir wget-sandbox && cd wget-sandbox
				
			

Now that we have a sandbox to experiment in we can safely run the command below, which uses -m or --mirror:

				
					wget -m https://rogerdudler.github.io/git-guide/
				
			

Be patient as some sites, depending on complexity, will take longer periods of time to complete.

wget2
Yikes! this is peak terminal spam!

Once the site is finished mirroring you’ll have a directory with the name of the site, and all its resources located neatly inside of it:

wget3
tree-ing the directory

Pretty radical right? In one command we can download an entire website (or at least the static version of it). Just for the sake of doing my duty, I must inform you that -m is short-hand for the following options:-r -N -l inf --no-remove-listing.

  • -r: recursively download files.
  • -N: time stamp the files.
  • -l inf: sets the maximum depth for files to be retrieved from
    their directories. By default the max is 5, but here we set it to infinity
    to mirror the site.
  • --no-remove-listing : this keeps FTP listing
    files, useful for debugging purposes.

While FTP is fine and dandy, from a security standpoint
please don’t use FTP, I only mention it here for the sake of completeness:

				
					wget sftp://demo:password@test.somewhere.net:22/
				
			

Parting ways

That was wget in a nutshell, another essential CLI command to try out! I can definitely see wget quickly becoming a webdev or archivist’s best friend with its mirroring feature. Imagine the scripting possibilities! That alone is worth taking a crack at wget a file or two. As always thanks for stopping by! Now go forth and download!

Meet the Author

Leave a Reply