the Patch-Tag blog


$ shuf /usr/share/dict/words | head -4 | tr '\n' ' '; echo
blotches rarity's unwieldier disarrange


If I got the math right, I believe we have here an easy MEMORABLE passphrase generator that should be relatively secure even against a distributed botnet password crack attack. Specifically, the password should resist a 1K botnet attack for 39 years, or a 1M botnet attack for 14 days.

Note that this is more secure than, say, a passphrase based on a lyric from a favorite song or some snip of text from a blog post, because the passphrase here is random.

Still a lot more memorable than a string of gobbledygook text.

Source / Explanation from my bashrc file:


thartman@pampasgrass:~/shellenv/shared>type thartman_password_gen
thartman_password_gen is a function
thartman_password_gen ()
{
echo ' comic explaining password strength in an intuitive way: http://xkcd.com/936
wc -l /usr/dict/words => ~ 100k
log 2 100k => ~ 16
echo 16 * 4 -> 64
distributed password cracking with a botnet: http://www.turnkeylinux.org/blog/tklbam-backup-passphrase
echo search 42 bits with 1000 computers => ~ 5 minutes
echo 64 - 42 => 22
echo search 64 bits with 1k botnet => (2^22 * 5) / (60*24 * 365) => ~ 39 years
secure password: ';
shuf /usr/share/dict/words | head -4 | tr '\n' ' '; echo
}

I started patch-tag coming on two years ago, and a blog post has been brewing about what went right, what went wrong, what I didn’t expect (or what was harder than I thought) and where I see things going. Here goes.

UPDATE: After I posted the initial version of this communique, a couple of people have asked if patch-tag is here to stay. Short answer. Yes Long answer, see my reply to Eric Kow below.

What went right.

Haskell is a joy to work with, the haskell community continues to be friendly for the newbie and fun for the expert, better now than when I started. The explosion of creativity on hackage has kept making simple things simpler while pushing the limits of what you can experiment with, with a simple library include.

Darcs worked out too. It is still my favorite versioning system and I use patch-tag for offsite versioning almost all my new work, and keep trying to convert my friends and cubicle mates to it. Darcs users are nice and a lot of them see to be working on interesting things.

Happstack as a server — overall I’m sold, though there are a lot of rough edges, and I watch snap and yesod with interest. With happstack too many monads and the documentation is sparse… but it got the job done for me, so ok. I really want to do a yesod test drive too — Michael Snoyman has put a lot of himself into this framework, and from browsing the docs and watching the intro videos I have to say he is doing an awesome job.

View Code: HSP/HStringTemplate.

HSP is a bit of an odd one becuase I don’t use HSP in patch-tag, but I use it for my other happstack work, and I like it. It isn’t all win. If you are using HSP you have to recompile every time you change the view code, and the time lag can get noticeable, particularly if you have monolithic View files with a lot of code, and rely on cabal install. HSP also relies on the trhsx preprocessor which means that projects that require it won’t cabal install ootb without ~/.cabal/bin
(or similar) is in your PATH. And there are some mystifying error mesages. I think it also uses template haskell, so slightly slower compiles. But even so, I like it. I like having the compiler check that my tags match up, I like being able to copy html from the wild directly into my view code, and I like that designers find it easy to work with.

HStringtemplate is what I use in patch-tag and I like that too. The compiler won’t check my html tags match up, but it is fast, no recompile is required, and it is pretty smart. I will probably depend on hsp in the future, and might even switch patch-tag to hsp at some point, but overall I have to say hstringtemplate worked out pretty well.

Linode/amazon cloud. I use linode to host patch-tag, and the aws cloud to occasionally spin up a dev server and experiment. Odd combination perhaps but so far so good.

Help from unlikely places. At different times and in different ways Matt Elder, Dan Patterson, and Ram Durbha had a major impact in moving patch-tag forward, for no tangible material reward. I hope to get more detailed in another post, but for now I would just like to say thanks. These guys taught me, maybe the most important lesson I learned from patch-tag: that the universe is basicaly a friendly place.

What went wrong.

The hyphenated name. Mama mia. Why didn’t I just call it patchtag? The actual reason is that I showed logo mockups with and without hyphens to a couple of friends and everybody seemed to like the hyphen version better. If I could go back in time and do-over, I’d probably ditch the hyphen.

The macid data store. I will probably be switching patch-tag back to some more traditional data store, probably a vanilla db. No one big reason, just a lot of small ones.
Well there is one big one. Patch-Tag app state sometimes just loses data. So, a new user account will get created and then vanish. I think this happens when the out-of-memory killer kills patch-tag before an event has been written to the macid serialization log. Honestly though, I’m not sure. Patch-tag can soldier with a bit of human intervention because the most valuable artifacts (repositories and namespaces) are on the file system and not in macid. Still not good.
Then there are the minor issues with macid. Small user / dev community. The documentation just isn’t all that great, and not that many people know how to use it. It was started by the happstack originators (alex jacobsen and co I suppose), and found a home in the current happstack cabal — kudos to jeremy shaw and the other regulars on the happstack list. But it just somehow never seems to have “clicked” as a technology from my perspective. It seems to me, somewhat impressionistically I admit, to use an inordinate amount of memory, and slow down compile time — all those template haskell directives. And I don’t really understand it all that well even after using it for a long time. Finally, macid isn’t as easy to roll back on corruption as I had initially thought — you have to edit a binary file and… ugh. Macid *wants* to be a silver bullet kind of nosql solution that will scale to multiple servers transparently, use native data structures, and make life easy for the developer. It is almost that, but it isn’t there yet and other
nosql solutions have gotten a lot of traction in the meantime.
Finally… it is somewhat painful to admit, but patch-tag really doesn’t need transactions. I probably could have just used text files and read/show serialization for state. What the heck was I thinking? I can’t even remember anymore.

Cabal/dependency management. I was a bit conflicted on whether to put cabal under the things that went wrong category because I have no intention of giving up cabal and it is clearly a core technology for this kind of project. That said, dependency wrangling proved to be a continuing time and energy drain throughout the project. On a nearly daily basis it seems, things would install one day and wouldn’t the next because somebody had updated a dependency on hackage and this broke something upstream. It doesn’t seem to happen that often in small to medium size haskell codebases, but projects that have a lot of moving parts (eg use happstack) appear more vulnerable to this problem. I suspect yi, gitit, gtk2hs have a similar installer experience.
There is a second thing about cabal that actually has an easy fix, but I didn’t discover for a long time. (Thanks Matt Brown for pointing this out!) Cabal install, which was how I compiled for the first year or so, started taking a long time. Like, over a minute. The fix is to just compile with ghc –make, after using cabal install just the first time to pull in all dependencies, and periodically to get updates. This seems to usually go about 4 times as fast, and when trying to stay in flow mode 15 seconds versus over a minute can start making a big difference.

Gitit fork. Gitit on the whole, I am happy with. I don’t think that many patch-tag users have taken advantage of this feature, including myself, but I really like the portability of keeping documentation alongside repo. Cross polination with another major happstack consumer is another win. Where I have regrets is having forked the gitit code somewhere along the line to get some look and feel features I wanted because I was in a hurry. Now gitit mainline has progressed several versions and I have to undo the fork I did and clear up all these niggling details to sync up, or maintain the fork forever. Whoops.

Unix security model alongside app security model to get ssh working. This is a mess for portability. If I want to switch servers I need to recreate system accounts, ssh keys, it’s a real drag. Alex Suraci’s Darcsden has an ssh server built in, written in native haskell, and apparently this makes it a lot easier to maintain. Transitioning patch-tag to this library and ditching the unix security details is a high priority for me in terms of future maintainability.

(Eventual) Moneti$ation. I thought I would go after darcs first, add git/svn other systems once darcs was solid, which would be a big enough market to make a sustainable business. Turned out not to be so easy!
On several occasions I was about to do payments, but then I got distracted fixing something without which it wouldn’t have felt right to start charging. Looking back on it, I kind of wish I had already gone after paying customers by this point.
The good news for me on a personal level is that having done patch-tag has really helped me get the kind of work I want to have, and advocate for using the technology I like, namely functional programming.

*********************

End ramble down memory lane, and announcing open source.

Patch-Tag source is open source at

http://patch-tag.com/r/tphyahoo/patch-tag-public

The install is non trivial, so for anybody that wants to check it
out without jumping through a whole lot of hoops (highly recommended) there is an amazon ami. Find the ami code by getting your amazon key, setting up ec2 command line tools and executing

thartman@ubuntu:~>ec2-describe-images -a | grep -i patchtag
IMAGE ami-febe5597 072945664613/patchtag dev
ami 072945664613 available public i386 machine aki-5f15f636 ari-d5709dbc ebs

You can run this ami from the comand line with ec2-run-instances, or use the aws gui or any of the numerous third party guis. Once you’re in, cd to the patch-tag directory, darcs pull the latest code, cabal install, and you should be good to go.

The main reason I am open sourcing patch-tag is to stimulate the haskell web-devel ecosystem by putting a “real world ready” app out into the open.

If anybody playing with the opened patch-tag wants to help me out with the project, here are my highest immediate priorities.

* Help me choose an open source license for patch-tag. I am considering gnu, bsd3, and CPAL (same license as reddit). What does the community think?

* Get patch-tag easier to install and on hackage. Patch-tag is not on hackage, because my feeling is that a program on hackage should Just Work or it violates legitimate expectations, and patch-tag cabal install is far from just working. Anybody is free to throw patch-tag on hackage under their own user account but I would say better to wait because the install issues *are* fixable with a little love and care.
Key subtodos for this are
* Unfork gitit and peg patch-tag to mainline gitit
* Use same ssh lib as darcsden so I can ditch most or all of the linux sysadmin stuff
* Make patch-tag machine instance for platforms other than ec2 (eg virtualbox, vmware, make it runnable on windows, etc).
* Diagnose and fix a suspected memory leak that results in regular visits from the dread oom killer.
* Transition patch-tag to a better understood / supported persistence solution than macid. My current thinking is sqlite, or possibly just text files with read/show serialization.
* Work on macid, and help realize the macid dream of straightforward nosql style persistence that uses native data structures for an easy start and scales to zillions of users with no unpleasant surprises.

Finally, and most importantly: Use haskell web-devel. Write documentation, ask questions, get on irc, and make the world a friendly place for people that like haskell and want to write the next facebook/ita/viaweb killer.

Thanks for tuning in, folks, and happy tagging!

For free.

Thanks, linode!

I have gotten a few private queries about backups lately, and thought it would be good to say a few words about what and how we keep your repositories safe.

Patch-tag is hosted on linode, and we do backups using S3 in the amazon cloud.

Patch-tag backups up all repository and application state data once a day using backup manager. Archive files, which are in dar format, are split in 25 megabyte chunks before uploading to S3. Backups are full, not incremental, so every day is a complete snapshot of all repositories along with application (happpstack) state.The policy is to keep at least one month of backups.

After backups complete, I get a daily email stating the size of the data loaded into S3, along the lines of

20100312: backups look ok
repos: 9.24Gb
application state: 0.80Gb

In case of errors creating archive files or uploading, I get a scary email saying that something went wrong. (Haven’t gotten one yet though.)

As a sanity check, I have done two full recreations of patch-tag from backups, using the EC2 amis I described in an earlier blog post. [ link please] I am happy to report there were no unpleasant surprises.

As of now, backups are sitewide. So you can’t access your personal deleted repos, but in case of emergency you could make a request for this data, assuming it is less than a month ago. We do have plans to change this in the near future to provide you more control over the backups of you repos.

If there are additional backup features you would like to see, please let me know.

Happy tagging!

Amazon EC2 is great, but it can get expensive, especially when you have an army of contractors doing all sorts of different things.

If you’re not careful, you can wind up creeping into a sort of tragedy of the commons situation where nobody shuts down little used EC2 instances because “that’s somebody else’s job” or simply because they’re not sure if anyone is using it. And since amazon bill doesn’t make it easy to see which server did what, let alone who was mostly using which server, it’s hard to formulate a good strategy for beating back the costs.

Here’s a simple, though somewhat radical solution. Prepay cloud developers for EC2 costs and have them use their own amazon accounts. The new improved amazon panel makes it easy to snapshot AMIs, do EBS volume management, and share these resources privately on a per-user basis. Share what you need to share, paypal the developer some funds and let them get on with it.

Suddenly your army of contractors is keeping a much more watchful eye on that EC2 balance because they don’t want to go back to their boss / client to request another prepaid-EC2 cash infusion more often than absolutely necessary.

This isn’t a silver bullet. If you are a fortune 500 company using “enterprise” cloud, you probably don’t want to start encouraging your sysadmins to start expensing basic server costs to their corporate credit card (actually, why not?…. nah). The prepay-EC2 strategy works best where you have a small well defined task that requires an EC2 environment, for example because setup is complicated and/or unautomated, and the developer is a cloudworker type outsourced contractor that wants to get coding with a minimum of friction. I think this approach could also work in a smallish startup environment casual enough that there is a degree of trust but using enouch computing resources that cost is beginning to matter. Prepaid EC2 might be just the trick for keeping the individual team members mindful of costs. I haven’t actually tried this though.

Minor detail: what do you do when the developer does such a good job minding costs that he completes the task far under the prepaid budget? I say, let them keep the excess!

Nota bene: the patch-tag production server is still in linode, and there are no concrete plans to switch to the cloud, though I am kicking the idea around.