Storage performance

I've been thinking about storage a lot lately, since I have to buy some pretty soon.

Did you know that there is a factor-of-40 difference in speed among common storage devices?

At the very slow end of the scale is any USB 2.0 device. These all top out at about 50 MB/s, the limiting factor being the speed of the USB 2.0 interface. If you have any USB 2.0 storage that you actually use, please upgrade now. Then we have FireWire devices at about 80 MB/s. Remember when FireWire used to be fast? Those days are gone.

Next up is a surprise: any NAS device. If you're using 100 megabit Ethernet then NAS is totally hopeless, but even with gigabit Ethernet you're limited by the Ethernet cable to about 100 MB/s. It doesn't matter how hot the NAS device itself is. You might think that NAS is great for storing photos and music and such (how many times have you heard "just install some drives in a Linux box and make a cheap NAS") but the performance is pretty poor.

Single hard drives connected to a SATA interface clock in around 100-200 MB/s. Hard drives are another "used to be fast" technology. The SATA interface itself goes up to about 600 MB/s, but the limiting factor is the hard drive's rotational speed and storage density. In general, higher capacity drives are faster than low-capacity drives, and drives with high rotational speeds are faster than drives with low rotational speeds. Fancy enterprise-class hard drives are faster than cheap laptop hard drives. But they're all running in the slow lane these days.

However, if you gather a bunch of hard drives together into an array, you can get some decent speed. With an array of four hard drives you can hit about 500 or 600 MB/s because you're reading and writing everything in parallel. Now you need to figure out how to hook the array up to your computer. You can use SATA (in eSATA form) or USB 3.0, both of which are pretty common.

For real speed, though, you need SSD. SSDs basically pick up where hard drives leave off; a slow SSD is like a fast hard drive. With an array of SSDs you can reach speeds of 1 GB/s or more. At these speeds, SATA and USB 3.0 don't cut it anymore. You need Thunderbolt, or better yet, Thunderbolt 2.

But even Thunderbolt 2 can't handle the fastest storage out there: PCIe SSDs. These are SSDs that connect directly to the PCIe (PCI Express) bus. Transfer speeds exceed 2 GB/s: Intel recently released a PCIe SSD that hits 2.8 GB/s. You can literally load an entire movie off this device in one second. There's little reason to buy a new computer without one of these. They're more expensive than hard drives, but not that expensive when you consider that they're ten times faster! And even if you have multiple terabytes of data, you probably don't need it all online at once. Buy a PCIe SSD for stuff you access the most and use a slower, cheaper drive for lesser-used stuff. Apple's "fusion drive" will do this automatically for you.

Comments

Post a Comment

Popular posts from this blog

UUIDs as primary keys

I wrote in my last post that I preferred to let the database just store data and and implement all the business logic in my Java code but that I usually let the database handle primary key generation. After thinking about it for a few days, I decided to experiment with using UUIDs as primary keys. Java 5 has a convenient java.util.UUID class that I used to replace the integer identifiers in my entity classes. I found a bunch of blog entries about using UUID versus integer primary keys. UUIDs do have some disadvantages: UUIDs are larger than integer IDs. A UUID, when stored in its most compact form, consumes 128 bits, or 16 bytes. That's four times larger than a regular 32-bit integer. UUIDs are binary. While they fit in a 16-byte binary column, when expanded into a string, they typically consume 32 or 36 characters, depending on whether the string is the human-friendly 00112233-4455-6677-8899-aabbccddeeff format. UUIDs are computationally harder to generate than integers. The disad
Read more

Scala and Kotlin

I've been spending the last three years writing code in Scala. In general I've been pretty happy with the language. It's much more compact than Java, and its higher-level features, such as closures (which have been around forever but which were popularized, I think, by Ruby), enable me to discard several of the Java tools and frameworks that I've had to drag around. Scala also comes with a very powerful collection library that I use on just about every line of code. Lately I've been following Kotlin, which looks very promising. I'm old enough to realize that this may just be a case of the grass looking greener on the other side of the fence. Every programming language has its flaws, and Scala is no exception. There's always the sense that maybe the next language will be The One. The One that does everything perfectly. The One that incorporates all that is good about Java and Scala and C# and C and C++ ( well okay, maybe not C++ ), introduces exciti
Read more

Fighting words on Ruby

Is anyone but me dismayed by the current state of Ruby? It's a great language, and it has a lot of promise--but there are also major issues that make Ruby and related efforts seem more like a set of academic projects than serious production-quality programming tools. The main thing that prevents me from using Ruby is that the 1.8 interpreter is too weak, and the other alternatives are too immature. I can use Ruby for scripting in the same way that I'd use Perl or PHP, but I wouldn't want to write a major application using a technology that doesn't support native threads, Unicode, or some kind of GUI. Yes, I know that you can roll some primitive Unicode support into Ruby, and I know that there are several GUI toolkits out there, but overall, Ruby's support for anything beyond basic scripting is spotty. So what are the alternatives? I guess the front-runner is JRuby, but JRuby is really just a scripting language that works with Java, not something that you'd
Read more