Comments on: Fast file rename and copy http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/ Reliable online storage powered by Amazon S3 Sun, 07 Sep 2008 18:48:01 +0000 http://wordpress.org/?v=2.3.1 By: Jackal http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-109 Jackal Mon, 02 Apr 2007 08:49:24 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-109 Well, I had my ISP conduct some tests from their headquarters. On the cable modem platform, they found the same odd 720kbps limit (so I know it's not my computer). But on their corporate network (presumably directly connected to their backbone), they got speeds of 5+mbps. So it's not Amazon. (The techs at my ISP couldn't figure out what was causing the limit, but they weren't really motivated to solve it, I guess...) But that's not why I'm posting. I finally got my invite to EC2, and I loaded up a test image and played around with it. Fun--but more expensive for my type of usage than I thought. I was under the impression that I would be charged for hours of CPU time--in other words, since my instance would most likely remain mostly idle, I would only be charged at most for an hour or two of usage per month. After I played with it and read the terms, I understood that I would be charged for each hour the machine remained up, regardless of usage (an instance-hour, not a CPU hour--so, basically, it's a virtual dedicated server, not really shared hosting). So, leaving a machine up for me to occasionally connect to and play with would cost me $73 per month--not a good bargain for me, although that seems like a fairly good deal for dedicated hosting (of course, data transfer is extra). Actually, the reason I post all that here is because I didn't realize it, but JungleDave's paying that $73 per month just to provide us with this very convenient fast-renaming service! (Well, maybe he's using it for other things, too...) I hope he's able to cover that with JD subscription sales...! Well, I had my ISP conduct some tests from their headquarters. On the cable modem platform, they found the same odd 720kbps limit (so I know it’s not my computer). But on their corporate network (presumably directly connected to their backbone), they got speeds of 5+mbps. So it’s not Amazon. (The techs at my ISP couldn’t figure out what was causing the limit, but they weren’t really motivated to solve it, I guess…)

But that’s not why I’m posting. I finally got my invite to EC2, and I loaded up a test image and played around with it. Fun–but more expensive for my type of usage than I thought.

I was under the impression that I would be charged for hours of CPU time–in other words, since my instance would most likely remain mostly idle, I would only be charged at most for an hour or two of usage per month. After I played with it and read the terms, I understood that I would be charged for each hour the machine remained up, regardless of usage (an instance-hour, not a CPU hour–so, basically, it’s a virtual dedicated server, not really shared hosting). So, leaving a machine up for me to occasionally connect to and play with would cost me $73 per month–not a good bargain for me, although that seems like a fairly good deal for dedicated hosting (of course, data transfer is extra).

Actually, the reason I post all that here is because I didn’t realize it, but JungleDave’s paying that $73 per month just to provide us with this very convenient fast-renaming service! (Well, maybe he’s using it for other things, too…) I hope he’s able to cover that with JD subscription sales…!

]]> By: Jungle Dave http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-108 Jungle Dave Wed, 28 Feb 2007 16:44:01 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-108 The speeds are now listed in kilobits/sec, so 10000-30000kbps corresponds to 10-30mbits/sec. I've seen it go as high as 70mbit/sec on a single large file. When writing the data, it actually goes to multiple datacenters at the same time, so that speed is actually quite good. The speeds are now listed in kilobits/sec, so 10000-30000kbps corresponds to 10-30mbits/sec. I’ve seen it go as high as 70mbit/sec on a single large file. When writing the data, it actually goes to multiple datacenters at the same time, so that speed is actually quite good.

]]> By: Jackal http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-102 Jackal Wed, 28 Feb 2007 05:43:59 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-102 OK, sorry for the third post in a row, but... I just did some JungleDisk sorting (moving some directories into a more organized hierarchy). I'm seeing copy rates of anywhere between 10000 kbps to 30000 kbps (roughly 1-3 MBps) on large files. That's A LOT better than downloading and reuploading the files, so thanks for implementing this, Dave! Still, it's a bit slower than I would expect for computers in presumably the same data center as the storage (and presumably using at least 100Mbps NICs if not 1000Mbps NICs). Anyone getting better copy speeds? Should I expect better speeds? I've been hashing out some slower-than-expected data transfer rates with S3 on the AWS forums. On my 3mbps home connection, I can only download at ~720Kbps (and it's always exactly 720Kbps, so it almost seems as if there's a packet shaper or some cap somewhere--even though no other downloads from other servers are capped at this speed, and even though I'm only 12-15 hops away from their Seattle POP, via my ISP's OC192 fiber optic cable directly between Anchorage and Seattle), and at my university, I get bad speeds, even factoring in the 5mbps limit on the packet shaper and the OC3 we have to the commodity Internet. But that's not really related to this post...I'm just hoping that Amazon increases the performance of their networks... OK, sorry for the third post in a row, but…

I just did some JungleDisk sorting (moving some directories into a more organized hierarchy). I’m seeing copy rates of anywhere between 10000 kbps to 30000 kbps (roughly 1-3 MBps) on large files. That’s A LOT better than downloading and reuploading the files, so thanks for implementing this, Dave!

Still, it’s a bit slower than I would expect for computers in presumably the same data center as the storage (and presumably using at least 100Mbps NICs if not 1000Mbps NICs). Anyone getting better copy speeds? Should I expect better speeds?

I’ve been hashing out some slower-than-expected data transfer rates with S3 on the AWS forums. On my 3mbps home connection, I can only download at ~720Kbps (and it’s always exactly 720Kbps, so it almost seems as if there’s a packet shaper or some cap somewhere–even though no other downloads from other servers are capped at this speed, and even though I’m only 12-15 hops away from their Seattle POP, via my ISP’s OC192 fiber optic cable directly between Anchorage and Seattle), and at my university, I get bad speeds, even factoring in the 5mbps limit on the packet shaper and the OC3 we have to the commodity Internet. But that’s not really related to this post…I’m just hoping that Amazon increases the performance of their networks…

]]> By: Jackal http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-101 Jackal Wed, 28 Feb 2007 04:53:26 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-101 Oops, helps if I actually read your blog in total (instead of just this entry). I see your proposed pricing scheme post--sounds like a winner! Oops, helps if I actually read your blog in total (instead of just this entry). I see your proposed pricing scheme post–sounds like a winner!

]]> By: Jackal http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-100 Jackal Wed, 28 Feb 2007 04:52:02 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-100 I was planning on getting an EC2 account to do the same thing with my other S3 site (I use Interarchy to publish publicly-downloadable files), and I'm happy to see that you've come up with the same solution! My only question is: since it's your EC2 account, you're paying for the EC2 compute time (which shouldn't be a lot for a simple mv or cp function)--so how are you going to pay for all of this? It seems generous for you to pay out of your pocket for it, but (unless you've inherited more money than you know what to do with) that's not a long-term solution. Do you eventually plan to start charging for JungleDisk? (I love the utility and love even more that it's free, but I *would* gladly pay a small sum for it...) I was planning on getting an EC2 account to do the same thing with my other S3 site (I use Interarchy to publish publicly-downloadable files), and I’m happy to see that you’ve come up with the same solution!

My only question is: since it’s your EC2 account, you’re paying for the EC2 compute time (which shouldn’t be a lot for a simple mv or cp function)–so how are you going to pay for all of this? It seems generous for you to pay out of your pocket for it, but (unless you’ve inherited more money than you know what to do with) that’s not a long-term solution. Do you eventually plan to start charging for JungleDisk? (I love the utility and love even more that it’s free, but I *would* gladly pay a small sum for it…)

]]> By: jd http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-107 jd Sat, 24 Feb 2007 14:56:03 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-107 Thanks for the reply Dave. The signing feature sounds cool. I am in line to get an EC2 account and look forward to working with them down the road. It really looks like the Amazon guys have thought this through really well. Congrats on great software, BTW. I enjoy using it and I like your proposed pricing scheme. Good work. -JD Thanks for the reply Dave. The signing feature sounds cool. I am in line to get an EC2 account and look forward to working with them down the road. It really looks like the Amazon guys have thought this through really well.

Congrats on great software, BTW. I enjoy using it and I like your proposed pricing scheme. Good work.

-JD

]]> By: Jungle Dave http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-106 Jungle Dave Sat, 24 Feb 2007 02:46:44 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-106 Sure - it's actually quite simple. S3 allows you to "sign" a request on one machine, then execute it on another. Jungle Disk simply signs the requests locally, then sends them to the EC2 server to execute. Signed requests are only valid for that specific operation, and only for a short period of time. Sure - it’s actually quite simple. S3 allows you to “sign” a request on one machine, then execute it on another. Jungle Disk simply signs the requests locally, then sends them to the EC2 server to execute. Signed requests are only valid for that specific operation, and only for a short period of time.

]]> By: jd http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-105 jd Fri, 23 Feb 2007 17:19:38 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-105 Dave, Great use of EC2! I'm a tad confused, however, of how you can set up an EC2 server to rename files in a user account without transferring the user's secret key. Does Jungle Disk just transfer the user id and password to the EC2 machine to get authority to change the file name? Any light you could shed on how this is accomplished would be greatly appreciated! Thanks, JD Dave,
Great use of EC2! I’m a tad confused, however, of how you can set up an EC2 server to rename files in a user account without transferring the user’s secret key. Does Jungle Disk just transfer the user id and password to the EC2 machine to get authority to change the file name?

Any light you could shed on how this is accomplished would be greatly appreciated!

Thanks,

JD

]]> By: Colin Henderson http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-104 Colin Henderson Fri, 23 Feb 2007 05:45:45 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-104 Fantastic job. The only thing you guys need is more promoters. This is the simplest, smoothest back up solution. Colin Fantastic job. The only thing you guys need is more promoters. This is the simplest, smoothest back up solution.
Colin

]]> By: Avi Flax http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-103 Avi Flax Thu, 22 Feb 2007 22:23:12 +0000 http://blog.jungledisk.com/2007/02/22/fast-file-rename-and-copy/#comment-103 Sounds impressive, looking forward to it! Sounds impressive, looking forward to it!

]]>