In UNIX and Linux-based systems, the wget command is a powerful utility for downloading files from the internet. Its versatility allows it to handle various tasks, from downloading single files to mirroring entire websites. This guide aims to provide an in-depth exploration of wget’s full potential.
Deciphering the Basics of Wget
Wget is a non-interactive, free utility designed for downloading files from the web. It supports HTTP, HTTPS, and FTP protocols, and can retrieve files through HTTP proxies.
The basic syntax of wget is as follows:
wget [option] [URL]Within the above syntax:
- The command
wgetis the trigger for deploying the utility. [option]acts as a discretionary parameter enabling you to define various configurations. Wget comes with a vast array of options that allow users to modify its operation. From enabling background downloads to limiting download speeds, a wide assortment of settings can be fine-tuned to suit your needs.[URL]signifies the web address of the file you are aiming to download. This is where you input the specific location on the web from where the utility fetches the file.
Detailed Wget Command Examples
1. Downloading a Single File
The most straightforward use of wget is to download a single file from a specified URL.
wget https://example.com/file.zipThis command will download the file named file.zip from example.com and save it in the current working directory.
2. Downloading in the Background
For downloading large files, wget can be run in the background using the ‘-b’ option.
wget -b https://example.com/largefile.zipThis command initiates the download of largefile.zip in the background, allowing you to continue using the terminal for other tasks.
3. Limiting Download Speed
To prevent wget from consuming all available bandwidth, you can limit the download speed using the ‘–limit-rate’ option.
wget --limit-rate=200k https://example.com/file.zipThis command restricts the download speed to 200 KB/s, ensuring other online activities are not disrupted.
4. Downloading Multiple Files
Wget can download multiple files at once. This is achieved by creating a text file with a list of URLs, and using the ‘-i’ option.
wget -i urls.txtIn this example, urls.txt is a text file containing the URLs of all the files to be downloaded. The command downloads all the files listed in urls.txt.
5. Downloading an Entire Website
Wget can be used to download an entire website for offline viewing. This is done using the ‘–mirror’ or ‘-m’ option.
wget --mirror https://example.comThis command mirrors the entire example.com website, downloading all its pages and assets for offline use.
6. Downloading Files with Specific Extensions
To download files with specific extensions from a website, use the ‘-r’ option for recursive download, ‘-A’ to specify the file types.
wget -r -A.pdf https://example.comThis command downloads all PDF files from example.com and its subdirectories.
7. Downloading Files Under Different Names
Using the ‘- O’ option, you can save a file under a different name.
wget -O myFile.zip https://example.com/file.zipThis command downloads file.zip from example.com and saves it as myFile.zip in the current directory.
8. Saving Files in a Specified Directory
You can specify a directory to save the downloaded file using the ‘-P’ option.
wget -P /path/to/directory https://example.com/file.zipThis command downloads file.zip from example.com and saves it in the directory specified by /path/to/directory.
9. Setting Retry Attempts
You can set the number of retry attempts in case the download fails using the ‘–tries’ option.
wget --tries=3 https://example.com/file.zipThis command attempts to download file.zip from example.com up to 3 times before giving up.
10. Downloading via FTP
Wget can also download files via FTP. You need to specify the username and password.
wget --ftp-user=USERNAME --ftp-password=PASSWORD ftp://example.com/file.zip
This command downloads file.zip from the FTP server at example.com using the specified username and password.
11. Continuing Interrupted Downloads
If your download gets interrupted, you can continue it using the ‘-c’ option.
wget -c https://example.com/file.zipThis command resumes the download of file.zip from example.com from where it left off.
12. Retrieving Whole Websites
You can download the entire content of a website for offline viewing using a combination of options.
wget --mirror --convert-links --page-requisites --no-parent -P /path/to/directory https://example.comThis command downloads the entire example.com website and its assets, converts the links for offline use, and saves everything in the directory specified by /path/to/directory.
13. Locating Broken Links
Wget can be used to find all broken URLs that display a 404 error on a specific website.
wget -o wget-log -r -l 5 --spider http://example.comThis command checks all links on example.com up to 5 levels deep and logs the results in wget-log.
14. Downloading Numbered Files
If you have files or images numbered in a specific list, you can easily download all of them.
wget http://example.com/images/{1..50}.jpgThis command downloads the images 1.jpg to 50.jpg from example.com/images.
15. Downloading Files with a Timeout
With the ‘– timeout ‘ option, you can set a timeout for wget to stop trying to connect or read from a site.
wget --timeout=30 https://example.com/file.zipThis command attempts to download file.zip from example.com, but will stop trying if it cannot connect or read from the site within 30 seconds.
16. Ignoring Case When Matching Files/Directories
The ‘-nc’ or ‘–no-clobber’ option prevents wget from overwriting existing files or directories.
wget -nc https://example.com/file.zipThis command downloads file.zip from example.com only if a file with the same name does not already exist in the current directory.
17. Downloading Only Certain File Types
You can instruct wget to download only specific file types from a site using the ‘-r’ and ‘-A’ options.
wget -r -A jpeg,jpg,bmp,gif,png https://example.comThis command downloads all JPEG, JPG, BMP, GIF, and PNG files from example.com and its subdirectories.
18. Downloading All Files Except Certain File Types
You can instruct wget to download all files except certain file types from a site using the ‘-R’ or ‘–reject’ option.
wget -r -R pdf https://example.comThis command downloads all files from example.com and its subdirectories, except for PDF files.
19. Downloading Files from Websites That Use Cookies
Some websites require users to accept cookies before they can access files. You can use the ‘–load-cookies’ option in this case.
wget --load-cookies /path/to/cookies.txt https://example.comThis command downloads files from example.com using the cookies stored in cookies.txt.
20. Downloading Files from Password Protected Sites
If a site requires username and password authentication, you can pass them using the ‘–user’ and ‘–password’ options.
wget --user=USERNAME --password=PASSWORD https://example.comThis command downloads files from example.com using the specified username and password.
21. Downloading Files from a List of URLs
You can download files from a list of URLs using the ‘-i’ option.
wget -i list.txtThis command downloads files from the URLs listed in list.txt.
22. Downloading Files to a Specific Directory
Using the ‘- P ‘ option, you can specify the directory where the downloaded files will be saved.
wget -P /path/to/directory -i list.txtThis command downloads files from the URLs listed in list.txt and saves them in the directory specified by /path/to/directory.
23. Downloading Files with a Specific File Extension
You can download files with a specific file extension using the ‘-A’ option.
wget -r -A .pdf https://example.comThis command downloads all PDF files from example.com.
24. Downloading Files from a Specific Directory
You can download files from a specific directory on a website using wget.
wget -r -np -nH --cut-dirs=3 -R index.html https://example.com/dir/subdir/This command downloads all files from the subdir directory on example.com.
25. Downloading Files with a Specific Pattern in Their URL
You can download files with a specific pattern in their URL using wget.
wget -r -np -nH --cut-dirs=3 -A '*.pdf' https://example.com/dir/subdir/This command downloads all PDF files from the subdir directory on example.com.
26. Downloading Files from a Website Using FTP
You can download files from a website using FTP with wget.
wget --ftp-user=USERNAME --ftp-password=PASSWORD ftp://example.com/
This command downloads files from example.com using FTP with the specified username and password.
27. Downloading Files from a Website Using HTTPS
You can download files from a website using HTTPS with wget.
wget https://example.com/file.zipThis command downloads file.zip from example.com using HTTPS.
28. Downloading Files from a Website Using HTTP
You can download files from a website using HTTP with wget.
wget http://example.com/file.zipThis command downloads file.zip from example.com using HTTP.
29. Downloading Files from a Website Using a Proxy
You can download files from a website using a proxy with wget.
wget -e use_proxy=yes -e http_proxy=PROXY_URL:PORT https://example.com/file.zip
This command downloads file.zip from example.com using the specified proxy.
30. Downloading Files from a Website and Saving Them with a Different Name
You can download files from a website and save them with a different name using wget.
wget -O newname.zip https://example.com/file.zipThis command downloads file.zip from example.com and saves it as newname.zip.
Conclusion
The wget command is a powerful tool with a wide range of applications. Its versatility makes it an essential tool for any user who frequently downloads files from the internet. This comprehensive guide gives you a deeper understanding of wget and its capabilities. Happy downloading!