lftp tricks: discover + queue

for ftp downloading, one of toughest problems we may face is job queueing and retry control. sometimes we may even expect to be able to modify the queue attributes such as specifying a different download account. i figured out the other day that, in fact, lftp is able to provide a flexible queue control to meet all these needs.

you need two lftp processes for discovering resources and queueing jobs: lftp1 and lftp2
1. to set retry parameters in lftp1. by default, lftp has a 80211-like backoff mechanism. the interval delayed for the next try is multiplied by a value with the previous one. we may only need to make the delay constant, such as 30s.
set net:max-retries 0
set net:reconnect-interval-base 30
set net:reconnect-interval-multiplier 1
2. open an easy-to-connect account in lftp2, so as to discover the remote dir you decide to download
3. open the download site in lftp1, and queue jobs. since ftp_site is supposed to be difficult to connect, you should just press ctrl+c to stop the "open" command. but this command is necessary because it tells lftp that you will queue jobs on this site.
open ftp_site
<ctrl+c>
queue mirror remote_dir_you_discovered
(notes: you can do 2 and 3 more times to discover and download on other sites. )

ending: by using queue control of lftp, you will find it can do almost everything that windows ftp clients can do, and maybe even simpler.