Feeds:
Posts
Comments

Amazon EC2 is great, but it can get expensive, especially when you have an army of contractors doing all sorts of different things.

If you’re not careful, you can wind up creeping into a sort of tragedy of the commons situation where nobody shuts down little used EC2 instances because “that’s somebody else’s job” or simply because they’re not sure if anyone is using it. And since amazon bill doesn’t make it easy to see which server did what, let alone who was mostly using which server, it’s hard to formulate a good strategy for beating back the costs.

Here’s a simple, though somewhat radical solution. Prepay cloud developers for EC2 costs and have them use their own amazon accounts. The new improved amazon panel makes it easy to snapshot AMIs, do EBS volume management, and share these resources privately on a per-user basis. Share what you need to share, paypal the developer some funds and let them get on with it.

Suddenly your army of contractors is keeping a much more watchful eye on that EC2 balance because they don’t want to go back to their boss / client to request another prepaid-EC2 cash infusion more often than absolutely necessary.

This isn’t a silver bullet. If you are a fortune 500 company using “enterprise” cloud, you probably don’t want to start encouraging your sysadmins to start expensing basic server costs to their corporate credit card (actually, why not?…. nah). The prepay-EC2 strategy works best where you have a small well defined task that requires an EC2 environment, for example because setup is complicated and/or unautomated, and the developer is a cloudworker type outsourced contractor that wants to get coding with a minimum of friction. I think this approach could also work in a smallish startup environment casual enough that there is a degree of trust but using enouch computing resources that cost is beginning to matter. Prepaid EC2 might be just the trick for keeping the individual team members mindful of costs. I haven’t actually tried this though.

Minor detail: what do you do when the developer does such a good job minding costs that he completes the task far under the prepaid budget? I say, let them keep the excess!

Nota bene: the patch-tag production server is still in linode, and there are no concrete plans to switch to the cloud, though I am kicking the idea around.

The Happstack guestbook example application is the best way to learn how to learn HSP with happstack.

It is really cool, and HSP, haskell’s answer to php, has gotten really significantly better in the past few months.

The only thing is that it is ridiculously hard to find.

Hopefully that link three paragraphs back will help.

There is something eerie about the starbucks where I sit programming, around 7 pm. It closes at 9 pm, yet it feels closed already. Employees push mops around. The temperature drops. The light seems a little dimmer.

It is the witching hour.

Out of nowhere, the six or seven people left here start singing happy birthday to Berry, who has made us all many an espresso. We’re on pitch. Somebody is harmonizing, and we actually sound pretty good! Birthday girl is genuinely touched.

It’s a little warmer in here.

Patch-Tag uses linode, but I’ve decided to try out EC2 for development purposes, as the hourly pricing model is attractive for server instances that don’t need to persist, and… well, I’m curious.

First off, with a big bow of respect to the amz engineers, I’ll tell ya, EC2 has gotten a lot better lately.

The last time I used aws was 3 or so months ago when I was configuring cloud stuff for a video startup that used a lot of disk, in conjunction with rightscale management console. Everything worked from a technology standpoint, but my gut feeling vis a vis usability was that aws was an awful mess: guis that didn’t work right, command line interfaces with hard to find documentation, inconsistent flags… beh.

https://console.aws.amazon.com

is hugely improved, documentation and help seems to be better organized… everything is just just more polished… better.

Anyways, to get to the meat of my post, I have created two public amis that I hope may prove useful to developers and newbies alike.

EC2 Amazon Machine Image: ami-3122ce58

AMI ID: ami-3122ce58
Name: Haskell on EC2
Description: Based on elastic karmic, with haskell platform
Source: 072945664613/Haskell on EC2

and… a few cabal installs later

EC2 Amazon Machine Image: ami-3d23cf54

AMI ID: ami-3d23cf54
Name: Gitit Happstack Demo on EC2
Description: Gitit/Hapstack on EC2
Source: 072945664613/Gitit Happstack Demo on EC2

Now, these images are really not such a bit deal since what with cabal and the haskell platform, there have been huge gains in package installability. But they weren’t completely trivial either, due to missing dependencies, the quickcheck 1 to quickcheck 2 switch that broke a lot of happstack, and various other hoops I had to jump through…

*I* had to jump through.

*You* just load the AMIs.

I recommend to start here if you’re an aws newbie.

Happy hacking!

PS Check out the latest from Simon Marlow for another haskell on EC2 post


In case there any aws experts reading this, here is also a snip of a chat of me fumbling around. Feel free to comment or add your questions.


<patch-tag> I am creating an EBS bootable AMI, which takes (at least) > 10
minutes. Say five minutes after start I do “touch somefile”. Will
“somefile” be included in the new AMI? What governs this behavior?
<patch-tag> (I actually am performing this experiment at the moment but I am
curious as to what to expect and why)
<flaccid> um i havnt played with ebs boot yet, but i would assume so, its
meant to be persistent
<flaccid> try it out
<patch-tag> The reason I am asking is, say I want to snapshot while
configuring something, and I am at a safe/happy place. If the
snapshot applies to that exact moment in time, I can keep doing
dangerous things. But if the snapshot extends in time I have to
sit on my hands till it completes before continuing work.
<flaccid> well a snapshot is a snapshot
<patch-tag> er… ?
* flaccid looks up ebs boot
<patch-tag> not particular to ebs boot. question applies to any way of
creating amis.
<patch-tag> (sorry for obfuscating the issue)
<flaccid> oh right
<flaccid> well an AMI is indeed a snapshot
<flaccid> thus why it is called an image
<flaccid> looking at ebs boot, it does seem to be only the ability to boot
from ebs snapshots, so its not a persistent image but a manual way
of gaining persistence
<flaccid> and it costs more
<patch-tag> changing subject — costs more? I thought ebs storage when offline
was same as S3, 10c/G/month.
<patch-tag> also, AMI creation completed, so I’m actually going to check if
that file I created five minutes in got bundled…
<patch-tag> aaaaaand… it’s not there.
<patch-tag> I guess what’s getting snapshotted is ram in a certain state, not
hard disk. which is why it can be truly instantaneous.
<flaccid> everyone has said that

http://aws.amazon.com/about-aws/whats-new/2009/12/03/amazon-ec2-instances-now-can-boot-from-amazon-ebs/

works out costing more
<flaccid> patch-tag: where was the file on the filesystem you touched ?
<patch-tag> /home/ubuntu/testfile
<flaccid> afaik by default /home is not bundled
<flaccid> you will need to check that
<flaccid> and obviously ensure /home is actually on the root fs (/)
<patch-tag> no, it’s bundled. I know this from a previous snapshot.
<patch-tag> any insight on why boot from ebs costs more? I guess because if
you make a lot of snapshots those 10c/G/month add up?
<patch-tag> And the convenience of it makes it tempting to do more snapshots?
<flaccid> well if /home is bundled then, i have no idea. bundling is
bundling…
<flaccid> and yeah can’t answer your second question. unfortunately i’m not
amazon sorry
<patch-tag> just in case it’s not clear, we’re in good shape. this is the
behavior I want.
<patch-tag> My assumption is that snapshot is point in time, and I can
initiate dangerous configs after initiating snapshot without
fear.
<flaccid> like i was saying before, yes a snapshot is a snapshot
<flaccid> whether it is a bundled ami or ebs snapshot, they are both snapshots

Patch-tag is now https safe, so you can access your projects with cryptographic security. Just use https style urls when interacting with patch-tag.

By the way — for you code shops with mission critical, high-value, national security endangering repos hosted at patch-tag — https isn’t completely safe.

You could, for instance, get your password sniffed by a man in the middle attack with arp poisoning, if you are in a public network like an internet cafe, or on a trusted network with a coworker who enjoys hanky panky.

There is a simple thing you can do to foil the terrorists though — just click outside the text fields and type some gibberish when you are entering your password.

Ha! Take that script kiddies!

Please enjoy secure repos at patch tag, and as always…

Happy tagging.

************************

UPDATE: Please note the comments which contradict some of my claims. I think I may be wrong about some of this stuff, and am taking a backtrack to make sure my understanding is solid. Watch this space.

One of the great things about distributed version control is that you can unplug from the internet and keep checking in.

If you recreate your gitit wiki offline, you can unplug from the internet and keep interacting with your repository via your offline gitit wiki.

Here’s a newsgroup discussion that describes how to use your gitit wiki offline

Disclaimer: If you are not technically savvy, and using patch-tag only for the gitit wiki, this option is probably not for you. I have some ideas percolating though that might make this a lot more straightforward if having an offline gitit wiki is something people want.

Patch Tag has a bug tracker, an anonymously editable gitit wiki hosted at patch-tag.

Full details at the mailing list, where I made the announcement.

Follow

Get every new post delivered to your Inbox.