Introduction

wget is a Linux file downloader. It is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.


Use Cases

We'll go through a few examples on using wget.



Download a Single File


# wget http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso


This command will download the CentOS 7 ISO file in the user’s current working directory.



Download File and Save with a Different File Name


We can save the file we downloaded with a different filename. We can use the uppercase "-O" option.


# wget -O CentOS7.iso http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso



Resume Partially Downloaded File


There are instances where we were downloading a file but in the middle, the connection was interrupted.

Using the "-c" option with wget will allow us to continue where we left off.


# wget -c http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso


If you do not specify the "-c" option, wget will append a .X (where X is a number starting from 1) extension and consider it as a new download.


Download Multiple Files


To download multiple files using wget, you need to create a text file first and add the URLs in the file separated by a new line.


# cat download-list.txt
url1
url2
url3
url4


To download these files, use the following command:


# wget -i download-list.txt



Download Files in the Background


We can download files in the background using the "-b" option.


# wget -b http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso


# Continuing in background, pid 4505.
# Output will be written to ‘wget-log’.


As you can see, it says the output will be written to a file named "wget-log". This file will be in the current directory you executed wget from.



Limit Download Speed


By default, wget will attempt to use all available bandwidth. This can be an issue if you are using a shared connection. You can use the "--limit-rate" option to limit the bandwidth used by wget.


# wget --limit-rate=200k http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso


In this example, we are limiting the download speed to 200k.



Increase Retry Attempts


By default, wget only retries up to 20 times to make the download successful. This is useful in problematic connections and you are downloading a large file. Chances are there will be failure in downloading. Use the "--tries" option.


# wget --tries=75 http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso



Logging


We can have wget save its output to a logfile using the "-o" option.


# wget -o download.log http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso


This will save the output from wget to "download.log" in the user's current directory.




Downloading from Password Protected Sites



We can download from password protected sites using the "--http-user" and "--http-password" options. For FTP sites, replace "http" with "ftp".


# wget -- http-user=username --http-password=password download.log


http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso



Setting the User Agent


Sites may block the wget user agent from accessing. In this case, we can change the user agent by using the "-U" option. This allows an arbitrary string to be set for the user agent. 


# wget -U 'MyUserAgent' download.log http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso


A list of User Agents is available here.