Effective mirroring ftp sites using lftp
I’ve spent quite same time using different tools for mirroring files from one ftp server to another or locally. Some time ago I’ve discovered LFTP, and decided to use it instead of other alternatives I’ve used over time like:
- wget that is requiring high cpu usage and doesn’t support FTP as a destination
- rsync – because you can’t allways install rsync on the ftp server.
Pros
- high transfer speed
- low cpu-usage (wget will kill your cpu!)
- ftp-to-ftp support – both source and destination are FTP servers
- FXP support – this would direct copy ftp from one server to another without sending them through your computer, still the FTP server must support this FTP protocol extension
- parallel transfer support with advanced options
- resume support
- multiplatform – Linux, OS X, Windows
- easy to install -like “yum install lftp”, or “port install lftp”
- open-source
Cons
- no native Win32, currently you have to install Cygwin but that contains lftp
- the win32 version it’s not too stable on all systems I’ve tried it.
- just one developer maintaining it, but the project is active
Some hints on lftp usage
--excludeLog :exclude directories containing string Log--newer-than=now-5days: mirror only files newer than 5 days old--continue: resume incomplete transfers-e: remove files missing from the source
Include login and password into the URL like: ftp://user:password@ftp.acme.com/path/ but don’t forget to escape any special characters like /, $, .. from the username and password using url escaping (%xx).
English
I think if you combine fast UDP based file transfer with RSYNC capability you will save even more time. For example a commercial solution such as FileCatalyst will give you the best of both worlds.