Here’s a quick note on how to download the contents of an http directory via command line.

Using lftp to download an HTTP directory

Forget wget, lftp is what you covet!

What is lftp?

LFTP is a sophisticated ftp/http client, and a file transfer program supporting a number of network protocols. Like BASH, it has job control and uses the readline library for input. It has bookmarks, a built-in mirror command, and can transfer several files in parallel. It was designed with reliability in mind.

How do I use it to download the contents of an http directory?

sytax example #1:

$ lftp
lftp :~> mirror http: //example.org/somedirectory

sytax example #2:

$ lftp
lftp :~> mirror http://cans.cdm.depaul.edu/~badawi/courses/TDC368/codes/HW5/
Total: 1 directory, 6 files, 0 symlinks
New: 6 files, 0 symlinks
62538 bytes transferred
lftp :~> quit