In-My-Opinion.org

»How to FTP upload 1000+ files to a server«


How can someone upload a large amount of pages/files to a server?

If you have a huge number of files on your PC and want to put them online then uploading them file by file (= the usual way) is very time-consuming and error prone.

Here is my step by step solution:

1) Download Total Commander (you should have+use this prog anyway in your everyday's life) from


2) Zip all your files pressing ALT-F5 and choosing "TGZ" (this will create a .tgz file)

3) Upload the created .tgz file. Use any name you want. Uploading can be done with Total Commander, too.

4) Upload the following textfile as " untar.cgi " to the same location:

#!/bin/sh

echo "Content-type: text/html"
echo
echo "<html><head><meta http-equiv='Pragma' content='no-cache'>"
echo "<style type="text/css">"
echo "<--"
echo "div.outer { text-align:center; width: 100%; border-width:0px; padding-top:6px; padding-bottom:4px; padding-left:0px; padding-right:2px; }"
echo "div.inner { font-family:"courier new"; border:15px #a0a0a0 ridge; text-align:left; font-size:small; width:80%; background-color:black; color:#00DD34; padding:15; filter:progid:DXImageTransform.Microsoft.Gradient(GradientType=1, StartColorStr='#FF000000', EndColorStr='#aF000DD3'); }"
echo "a:link    { color:red; font-weight:bold; border-style: none;   text-decoration: none;  }"
echo "a:visited { color:red; font-weight:bold; border-style: none;   text-decoration: none;  }"
echo "a:active  { color:red; font-weight:bold; border-style: none;   text-decoration: none;  }"
echo "a:hover   { color:red; font-weight:bold; border-style: none;   text-decoration: underline;   }"
echo "-->"
echo "</style>"
echo "</head><body>"
echo "You currently have the following .tgz files:"
echo "<ul>"
for i in *.tgz
do
echo "<li>$i<br>" ;
echo "<div class='outer'><div class='inner'><pre>"
tar zxvf $i
echo "</pre></div></div>"
echo "</li>"
done

echo "</ul>"
echo "DONE!"

echo "</body></html>"



5) Change the atrributes of untar.cgi to 755 (otherwise you won't be able to start it). You can do this with Total Commander, too.

6) Run the cgi by typing
http://www.YourDomain.com/untar.cgi


Voila!

That's how I uploaded


Even if you have to upload only 30+ files this solution is better/faster than to do it file-by-file.


posted by knn
  

in-my-opinion.org -> Technology, Computers, Science, Internet -> Computers and Internet -> How to FTP upload 1000+ files to a server

Files UL Vs File UL+uncompress


I would like to see some numbers for how this is faster. Sure if you have 1000 text files and compress them, it will be a smaller item to ul but you still have to factor in decompression time.

Plus modern ftp client and server can support multiple connections so you can have 3+ files uling at the same time, making use of your bandwidth.

I am open to what you are saying and I believe I have uled a compressed file in the past instead of a bunch of files, but I would like to see some numbers that show this is faster and how much faster.

posted by volonteshiva
  

1000x 1 byte


volonteshiva:
I am open to what you are saying and I believe I have uled a compressed file in the past instead of a bunch of files, but I would like to see some numbers that show this is faster and how much faster.

Approx. 5x-10x faster.

It's like on your harddisk: If you have to copy 1000x a 1-byte-file it takes you much longer than 1x a 1000-byte file.


posted by knn
  

Nice Idea


This method is not faster 'cos its transferring less data, it's faster 'cos it reduces communication overheads.

If you transferred 1000 files, it would have 1000 x more;
• Open connection
• Transmit file details
• Set up transfer
• Close connection


Even with multiple connections, it still has to go through all the motions.

Granted there's the time to compress the files and uncompress them at the other end, but as with most things on the internet, the bottleneck is usually the transfer.

Besides, web pages compress really well

Using local file compression will always be better than the compression used by routers and other online kit, since it doesn't have to be real-time.

Shame I don't have a use for it. White laugh


posted by Marl64
  

screenshots?


I am still seeing alot of talk and no screenshots as evidence here.

posted by volonteshiva
  

There you have it. My test


volonteshiva:
I am still seeing alot of talk and no screenshots as evidence here.

You can try it yourself. Upload 100x 1byte and 1x100byte.

I did the test myself now. I uploaded 100 files, all smaller than 50 bytes (totalling 2254 bytes)

It took 2:50 minutes.

The same size as 1 file (2254 bytes) took 3 seconds.

Moreover there is yet another thing: Sometimes uploading stops because of time outs. But I noticed that the break downs take place often BETWEEN 2 files, that is
• Upload file 1
• Upload file 2
• Upload file 3
• break down
• File 4 could not be uploaded due to break down
• File 5 could not be uploaded due to break down
• File 6 could not be uploaded due to break down


If you upload 1 single file then the upload will be finished.
• Upload file 1 (the only file)
• break down (doesn't matter much, since everything has been uploaded).



posted by knn
  





Reply to topic






RegisterRegister
Log inLog in
 		The time now is 4 December 2008, 00:44
php B.B.