Measuring Cloud Performance — New Bitcurrent/Webmetrics Report

Hope everyone had a nice Fourth of July. Working closely with clients on very interesting projects is one of the best parts of my job in technology PR. Most of the time, the knowledge I gain is proprietary. But occasionally I get to share it, and today is one of those times.

Recently Bitcurrent conducted a cloud performance test commissioned by my client Webmetrics, a business unit of Neustar. Bitcurrent describes itself as part blog, part analyst firm and part resource site for the web community. The research project was led by Alistair Croll, a well-known name in the space and program chair for the Cloud Connect conference.

You can’t read a technology story today without the “cloud” being mentioned. But there is precious little reputable, empirical research by third parties to help customers understand the term, and to help determine which company they should go with for cloud services.

Bitcurrent has produced a fascinating report, tracking the cloud performance of the leading providers in a number of ways. The full 50+ page report is only available via registration off the Webmetrics web site, but I’d be happy to send an abridged version to anyone who’d like to see it. Just drop a comment here, send me an email or ping me on Twitter — @cparente.

Here is some background on the report. Five companies were analyzed:

1. Salesforce.com
2. Google App Engine
3. Terremark vCloud
4. Amazon Elastic Cloud Computing (EC2)
5. Rackspace Rackspace Cloud

Four different performance metrics were isolated:

• A simple web request for a small, static object — this tested responsiveness
• A request for a large, 2 Megabyte object — this tested network throughput
• A request that triggered one million mathematically intensive calculations — this tested computing power
• A request that triggered a search of 500,000 rows in a database for a string — this tested the back-end I/O (input/output)

I spoke at some length with Alistair about the report. He was jazzed about the report and said no one had ever measured the performance of cloud computing in this way. In particular, he stressed how the report uses histograms to illustrate the performance of each company. Histograms are a kind of graphical display that allowed for the display of the performance of all five companies for a specific request on a single graph. It’s as close as you can get to a distinct “fingerprint” for cloud performance:

Latency for delivery of 2 MB GIF

Alistair also had very nice words for Webmetrics underwriting the project and then staying completely hands off. This is from his blog:

At Bitcurrent, we try to remain nonpartisan. We want to contribute to the industry’s growth and understanding. So we were thrilled at Webmetrics/Neustar’s approach here: they financed the research into cloud performance, and let us use their monitoring tool to collect the data from our agents. They were supportive, and enthusiastic, but exercised no editorial control whatsoever over the content of the report. This kind of altruistic, good-for-the-market contribution is, in our opinion, the best kind of marketing, and we commend them for adopting it.

As I like to say, great work for great clients. If you’re currently evaluating how the cloud can help your business, this could be valuable information.  Ping me for a copy if interested.