i see bots

Buried in last week’s net neutrality proposal by the FCC was an interesting tidbit about the connected home.

As you may know, the controversial proposal put forth a requirement that all broadband providers allow their customers to access all (legal) online content. You are probably thinking, “I thought they were already required to do that??”

Actually, no. In April of this year, the FCC lost an important court ruling when the United States Court of Appeals for the District of Columbia ruled that the FCC lacked the authority to enforce the notion that providers could not discriminate against content services carried over their pipes. This case arose out of a finding the FCC issued in 2008 in which it ruled that Comcast had secretely blocked or slowed (“shaped”) traffic associated with the BitTorrent Service.

In May, the FCC chairman, Julius Genachowski, indicated in response that he was considering regulating broadband providers using the same regulatory powers the agency uses to regulate telephone networks. The difference is that broadband providers have been regulated as “information services”, not as “telecommunications services”; the latter brings with it much more scrutiny and regulation. This was a controversal proposal; service providers claimed that such a level of regulation would stifle the huge investments required to maintain and expand the country’s broadband networks.

However, in last week’s proposal, Genachowski did not take that significant step, but he did set forth a basic “no blocking” requirement while allowing providers to “shape” traffic as necessary to ensure network health, as long as providers provided notice of their practices. Wireless providers would be given more leeway, given the more limited nature of their networks. There was also a discussion about usage-based pricing, although this is less relevant, given that many major wired and wireless providers have implemented tiered or capped service levels.

(Personally, I found all of this ironic, given that in my neighborhood in Seattle, the best that the local telephone monopoly can offer me is dial-up networking. Comcast’s is the only wired broadband service available to me. So if I find that I don’t like Comcast’s price tiers or “shaping” practices, I really don’t have a choice. I suppose I could turn to 4G wireless services, but from what I can tell, those are still in the infancy stage, combining three or more of these unfortunate characteristics: expensive, relatively slow, limited to fairly low monthly data transfer ceilings, limited to certain geographic areas, and unreliable.)

I found these articles to be excellent summaries of the proposal: PC Magazine and New York Times.

I noticed the following statement in several of the articles on this topic:

“In addition, the proposal would let broadband providers experiment with routing traffic from specialized services such as smart energy grids and home security systems over dedicated networks, as long as the practice doesn’t slow down the public Internet.” (Yahoo!)

“The F.C.C. also will allow companies to experiment with the offering of so-called specialized services, providing separate highways outside the public Internet for specific uses like medical services or home security.” (New York Times)

What does this mean, I wonder? When I think about cloud-based home automation (see “A PIVOT TOWARDS THE CLOUD?” section in this post), or IP-based remote security monitoring, what I worry about isn’t latency but rather reliability. I wonder, then… will Comcast be offering me, say, some kind of uptime guarantee for a premium in their broadband offerings of the future?

What kind of broadband connectivity availability guarantee would be acceptable to me? Let’s say that they offer “three 9s” of availability, or 99.9%, which translates to just under 9 hours of downtime in a given year. (See a handy chart here)

“Three 9s” is the SLA (Service Level Agreement) that Amazon offers for its AWS offerings. If Amazon fails to meet that SLA, the customer is compensated in the form of a credit that appears on a future bill.

Would 9 hours of downtime a year be acceptable for a home automation system? I’d say that if those 9*3600=32,400 seconds of downtime were spread out uniformly across the year (say, 89 seconds each night between the hours of 2am and 3am!), I wouldn’t mind. But if all of those seconds of downtime hit all at once, it would be unacceptable.

To be fair, my current home PC-based home automation system has been down for hours at a time, usually planned but not always. A goal in moving to the cloud (all other things being equal) would be to leverage the actual professionals employed there to keep redundant systems healthy.

What about 99.99% availability? That’s just under a hour (53 minutes) a year. Much better, and probably even acceptable, but from what I can tell, only a few hosters offer that level of service, much less broadband providers.

As far as I know, the only services providers who venture into the rarefied air of “five 9s” - 99.999% - are the POTS guys - the telephone company. The infrastructure investment is impressive. I don’t think that anyone else, outside the corporate data center, bothers, though. Cloud-based architectures are designed to deal with 99.9%.

The possibility of significant disruptions in connectivity is the biggest objection as I think about the characteristics - positive and negative - of a cloud-based home automation or security monitoring service in the cloud. If there’s a possibility that the broadband connection could go down for hours at a time, the functions you’d offload to the cloud would end up being trivial or secondary at best.

Given that it’s unlikely that you could find a 99.99% availability SLA from a broadband provider, and even if you did, it would be very expensive, you might consider that continued thinking about a cloud-based approach for home automation would be a waste of time.

But there is another approach: consider implementing a “diverse” secondary broadband connection. For instance, if your main provider is Comcast, then you’d have a secondary emergency connection via, say, a 3G or 4G wireless provider (supplied by a provider other than Comcast). You would, in effect, be banking on the low probability that both providers would fail at the same time, and paying for that level of availability in the form of a second (wireless) connection that would rarely see service (and which, therefore, you hope would be inexpensive). Your connection to the cloud would be via a fast main pipe, backed up by a (perhaps cheaper) and slimmer backup pipe.

It’s probably too simplistic to compute the combined “SLA” of this primary/secondary connection architecture, but let’s do it anyway. Let’s assume that your primary provider is Comcast… I have anecdotal evidence that they don’t deliver at the 99.9% level… let’s call it it 99.7% (26 hours). Let’s assume that the wireless connection is slightly more reliably, given its roots as a telephone service - say, 99.99%. Therefore, the probability that both services would fail at the same time is something like: (26 hours/8766 hours in a year)  * (1 hour/8766 hours in a year) = .3% * .01 % = .003%. Are those acceptable odds?

Let’s assume that those are indeed reasonable odds. So if you’re happy about your bandwidth connection being reliable, what about the service you’re connecting to? If it’s hosted at Amazon AWS, then we’re back to the 99.9% level. But perhaps there are things we can do about that. More on this later.

(There are other critical objections to the concept of a home automation system in the cloud… I’ve found this thread on AVSForum to be interesting for this reason. The signal-to-noise ratio varies greatly over the 10 or so pages of this thread, but I found very interesting comments from a number of folks on that thread… but your mileage may vary.)

UPDATE #1, 6 December 2010: In the original version of this post, I said that “what I worry about isn’t latency but rather reliability”. Well, I found myself thinking about that statement a bit more, and decided to run some simple tests to figure out if that statement was warranted.

So I ran some very simple tests: I fired up an Amazon EC2 instance, telneted to it, and ran “ping” from there to my home gateway (a Linksys appliance), courtesy of Comcast’s home broadband offering (3 mb/s upload, I believe). I used a dynamic DNS service to resolve the address of my gateway. These are the results, via ping:

Amazon EC2 instance running in “us-east-1d” availability zone: ~92 ms

Amazon EC2 instance running in “us-east-1a” availability zone: ~99 ms

This compares with something like 2- to 2.9 ms ping times within my LAN (a PC on my LAN pinging my gateway).

So… the cloud is ~ 30 times ‘farther away’ as compared to my home’s LAN. Is this an issue?

Here’s a thought experiment: Let’s imagine that I’ve got Home Automation functions running on an EC2 instance. Let’s further imagine that a human in my house flips on a switch which has been automated in some way. How long would it take a simple Home Automation gateway on my LAN to notice that state change, communicate that fact to the EC2 instance, and send back a response? It would appear that cloud latency alone would require ~200ms alone for a round-trip. My very informal tests with Z-Wave (using HomeSeer’s “Z-Seer” utility) indicate that communications with a Z-Wave switch takes a small number of milliseconds. And if I assume that the code running in the EC2 instance can turn around a response in a small numver of milliseconds, then it would appear that the cloud latency times will dominate this scenario. Perhaps this cloud-based scenario could execute, end-to-end, in under, say, 300 ms. Would this be “interactive” enough for human-scale Home Automation? I don’t know. It might be.

I think the next test would be to actually code up a simple scenario that actually runs in the cloud and see how well it works… if anyone out there reading this has any ideas about this, would love to hear it…

UPDATE #2, 6 December 2010: Interestingly enough, saw these posts today:

• “Get Ready for $99 Security, Home Automation from Comcast/Xfinity” (link)
• “AT&T Acquires Xanboo, Developer of Home Automation Platform” (link)