Transferring lots of small files via FTP
April 17th, 2007 · Filed under ArchivesNote: For this article you need to have shell access or a way to unzip files in your hosting account.
FTP is one of the fastest file transfer protocols. This protocol is used for the transfer of files from one computer to another, using the Internet. FTP is usually used to transfer web documents and images for websites to the server where the website is hosted (read more about getting your website online).
When transferring your website to your server, it might take a while to upload, depending on the connection to the Internet you’re on. All files are transferred separately. Every time the transfer starts, a STOR command is sent to the server. This command initiates the transfer. The server responds with an acceptance or denial message. If accepted, the file is transferred. The server then sends a message back to the FTP-client to notify that the file is successfully transferred. If permissions need to be changed for the file, this will also be done in this process.
It’s also possible that the transfer does not reach full speed, because, before it can reach it’s maximum, the transfer of that single file is done.
Wordpress for instance has about 900 files. Think of what happens when this process needs to be repeated 900 times. There is time lost with the sending and receiving of server messages. For those on high speed connections this is not so much a problem, but those of us with connection speeds of 256K/128K down and 128K/64K up, every minute you can save is appreciated. The time it takes for a request to reach the server is about a quarter of a second on my 128K/64K download/upload connection. Each uploaded file sends and receives half a second worth of messages. That means that a transfer 900 files uses about 450 seconds for messages. 7 and a half minutes are added to the time needed to upload files.
Can it go faster?
One way to speed this process up a little is by zipping (compressing) all the separate small files into one big file, transfering it to your server and unzipping (decompressing) it on the server. This way messages are sent and received only once. Besides, the transfer can easily reach it’s maximum speed. Once the file is transferred, you need access to the server to decompress it.
For this process to work you need a way to decompress the uploaded file. One way this can be done is using the right commands in the shell (what is known as Command Prompt in Windows). Most hosting providers don’t allow shell access by default, but some of them can grant you shell access if you ask for it. In those cases they will probably ask you to fax or mail them a copy of your drivers license or some other form of identification. If you do get shell access, it’s as simple as unzipping the files in the right location. I will soon write an article about the zipping your web files on your local machine and unzipping them on your server.
cPanel, a control panel for hosting accounts, also has the feature of extracting (unzipping) files on the server. I know of cPanel, because that’s what I use. If your hosting company uses a different control panel, you may need to check for instructions on how to extract compressed files. You may need to ask your hosting provider if that feature is available.
Conclusion
If you have shell access to your hosting space, if you’re hosting control panel supports extraction of compressed files or if there’s another way to extract compressed files on your server, zipping your files into one big file, uploading the file and extracting the files from the compressed file may be a faster alternative to uploading small files one by one. This is especially true when you have a lot of small files.
Suck it
If you like the way I’m writing, check out the ways available to you to suck my feed.
Trackback URL for this post:
http://rehuel.com/2007/04/17/transferring-lots-of-small-files-via-ftp/trackback/




