Feeds:
Posts
Comments

Archive for the ‘Uncategorized’ Category


$ shuf /usr/share/dict/words | head -4 | tr '\n' ' '; echo
blotches rarity's unwieldier disarrange

If I got the math right, I believe we have here an easy MEMORABLE passphrase generator that should be relatively secure even against a distributed botnet password crack attack. Specifically, the password should resist a 1K botnet attack for 39 years, or a 1M botnet attack for 14 days.

Note that this is more secure than, say, a passphrase based on a lyric from a favorite song or some snip of text from a blog post, because the passphrase here is random.

Still a lot more memorable than a string of gobbledygook text.

Source / Explanation from my bashrc file:


thartman@pampasgrass:~/shellenv/shared>type thartman_password_gen
thartman_password_gen is a function
thartman_password_gen ()
{
echo ' comic explaining password strength in an intuitive way: http://xkcd.com/936
wc -l /usr/dict/words => ~ 100k
log 2 100k => ~ 16
echo 16 * 4 -> 64
distributed password cracking with a botnet: http://www.turnkeylinux.org/blog/tklbam-backup-passphrase
echo search 42 bits with 1000 computers => ~ 5 minutes
echo 64 - 42 => 22
echo search 64 bits with 1k botnet => (2^22 * 5) / (60*24 * 365) => ~ 39 years
secure password: ';
shuf /usr/share/dict/words | head -4 | tr '\n' ' '; echo
}

Read Full Post »

This is mostly for myself, but maybe the googlebot will pick it up and help some others.

Basically, patch-tag encourages https: browsing post log in because, well, it’s the right thing to do. (IMHO, https should be the default option for web browsing, and there is a school of thought about that, but I’m in too much hurry to track it down. Comments welcome ;) )

So, I bought my ssl cert from godaddy to make it possible. And it expired, and I couldn’t remember how to make it work again.

After a bit of mucking around, I chose “renew ssl cert” in godaddy, and paid their pound of flash. Downloaded a little zipped bundle patch-tag.com.zip from godaddy. Contained 2 .crt files, patch-tag.com.crt and gd_bundle.crt.

To get things using the new cert, I edited

/etc/stunnel/stunnel.pem

leaving the top portion (pk) unchanged, and swapping out the bottom portion (cert) with the contents of patch-tag.com.crt file from godaddy.

I then did /etc/init.d/stunnel4 restart

afaict, good to go.

Not sure what that other cert file (gd_bundle.crt) is for.

That’s all folks.

Happy tagging!

PS This page was also helpful for

Read Full Post »

I started patch-tag coming on two years ago, and a blog post has been brewing about what went right, what went wrong, what I didn’t expect (or what was harder than I thought) and where I see things going. Here goes.

UPDATE: After I posted the initial version of this communique, a couple of people have asked if patch-tag is here to stay. Short answer. Yes :) Long answer, see my reply to Eric Kow below.

What went right.

Haskell is a joy to work with, the haskell community continues to be friendly for the newbie and fun for the expert, better now than when I started. The explosion of creativity on hackage has kept making simple things simpler while pushing the limits of what you can experiment with, with a simple library include.

Darcs worked out too. It is still my favorite versioning system and I use patch-tag for offsite versioning almost all my new work, and keep trying to convert my friends and cubicle mates to it. Darcs users are nice and a lot of them see to be working on interesting things.

Happstack as a server — overall I’m sold, though there are a lot of rough edges, and I watch snap and yesod with interest. With happstack too many monads and the documentation is sparse… but it got the job done for me, so ok. I really want to do a yesod test drive too — Michael Snoyman has put a lot of himself into this framework, and from browsing the docs and watching the intro videos I have to say he is doing an awesome job.

View Code: HSP/HStringTemplate.

HSP is a bit of an odd one becuase I don’t use HSP in patch-tag, but I use it for my other happstack work, and I like it. It isn’t all win. If you are using HSP you have to recompile every time you change the view code, and the time lag can get noticeable, particularly if you have monolithic View files with a lot of code, and rely on cabal install. HSP also relies on the trhsx preprocessor which means that projects that require it won’t cabal install ootb without ~/.cabal/bin
(or similar) is in your PATH. And there are some mystifying error mesages. I think it also uses template haskell, so slightly slower compiles. But even so, I like it. I like having the compiler check that my tags match up, I like being able to copy html from the wild directly into my view code, and I like that designers find it easy to work with.

HStringtemplate is what I use in patch-tag and I like that too. The compiler won’t check my html tags match up, but it is fast, no recompile is required, and it is pretty smart. I will probably depend on hsp in the future, and might even switch patch-tag to hsp at some point, but overall I have to say hstringtemplate worked out pretty well.

Linode/amazon cloud. I use linode to host patch-tag, and the aws cloud to occasionally spin up a dev server and experiment. Odd combination perhaps but so far so good.

Help from unlikely places. At different times and in different ways Matt Elder, Dan Patterson, and Ram Durbha had a major impact in moving patch-tag forward, for no tangible material reward. I hope to get more detailed in another post, but for now I would just like to say thanks. These guys taught me, maybe the most important lesson I learned from patch-tag: that the universe is basicaly a friendly place.

What went wrong.

The hyphenated name. Mama mia. Why didn’t I just call it patchtag? The actual reason is that I showed logo mockups with and without hyphens to a couple of friends and everybody seemed to like the hyphen version better. If I could go back in time and do-over, I’d probably ditch the hyphen.

The macid data store. I will probably be switching patch-tag back to some more traditional data store, probably a vanilla db. No one big reason, just a lot of small ones.
Well there is one big one. Patch-Tag app state sometimes just loses data. So, a new user account will get created and then vanish. I think this happens when the out-of-memory killer kills patch-tag before an event has been written to the macid serialization log. Honestly though, I’m not sure. Patch-tag can soldier with a bit of human intervention because the most valuable artifacts (repositories and namespaces) are on the file system and not in macid. Still not good.
Then there are the minor issues with macid. Small user / dev community. The documentation just isn’t all that great, and not that many people know how to use it. It was started by the happstack originators (alex jacobsen and co I suppose), and found a home in the current happstack cabal — kudos to jeremy shaw and the other regulars on the happstack list. But it just somehow never seems to have “clicked” as a technology from my perspective. It seems to me, somewhat impressionistically I admit, to use an inordinate amount of memory, and slow down compile time — all those template haskell directives. And I don’t really understand it all that well even after using it for a long time. Finally, macid isn’t as easy to roll back on corruption as I had initially thought — you have to edit a binary file and… ugh. Macid *wants* to be a silver bullet kind of nosql solution that will scale to multiple servers transparently, use native data structures, and make life easy for the developer. It is almost that, but it isn’t there yet and other
nosql solutions have gotten a lot of traction in the meantime.
Finally… it is somewhat painful to admit, but patch-tag really doesn’t need transactions. I probably could have just used text files and read/show serialization for state. What the heck was I thinking? I can’t even remember anymore.

Cabal/dependency management. I was a bit conflicted on whether to put cabal under the things that went wrong category because I have no intention of giving up cabal and it is clearly a core technology for this kind of project. That said, dependency wrangling proved to be a continuing time and energy drain throughout the project. On a nearly daily basis it seems, things would install one day and wouldn’t the next because somebody had updated a dependency on hackage and this broke something upstream. It doesn’t seem to happen that often in small to medium size haskell codebases, but projects that have a lot of moving parts (eg use happstack) appear more vulnerable to this problem. I suspect yi, gitit, gtk2hs have a similar installer experience.
There is a second thing about cabal that actually has an easy fix, but I didn’t discover for a long time. (Thanks Matt Brown for pointing this out!) Cabal install, which was how I compiled for the first year or so, started taking a long time. Like, over a minute. The fix is to just compile with ghc –make, after using cabal install just the first time to pull in all dependencies, and periodically to get updates. This seems to usually go about 4 times as fast, and when trying to stay in flow mode 15 seconds versus over a minute can start making a big difference.

Gitit fork. Gitit on the whole, I am happy with. I don’t think that many patch-tag users have taken advantage of this feature, including myself, but I really like the portability of keeping documentation alongside repo. Cross polination with another major happstack consumer is another win. Where I have regrets is having forked the gitit code somewhere along the line to get some look and feel features I wanted because I was in a hurry. Now gitit mainline has progressed several versions and I have to undo the fork I did and clear up all these niggling details to sync up, or maintain the fork forever. Whoops.

Unix security model alongside app security model to get ssh working. This is a mess for portability. If I want to switch servers I need to recreate system accounts, ssh keys, it’s a real drag. Alex Suraci’s Darcsden has an ssh server built in, written in native haskell, and apparently this makes it a lot easier to maintain. Transitioning patch-tag to this library and ditching the unix security details is a high priority for me in terms of future maintainability.

(Eventual) Moneti$ation. I thought I would go after darcs first, add git/svn other systems once darcs was solid, which would be a big enough market to make a sustainable business. Turned out not to be so easy!
On several occasions I was about to do payments, but then I got distracted fixing something without which it wouldn’t have felt right to start charging. Looking back on it, I kind of wish I had already gone after paying customers by this point.
The good news for me on a personal level is that having done patch-tag has really helped me get the kind of work I want to have, and advocate for using the technology I like, namely functional programming.

*********************

End ramble down memory lane, and announcing open source.

Patch-Tag source is open source at

http://patch-tag.com/r/tphyahoo/patch-tag-public

The install is non trivial, so for anybody that wants to check it
out without jumping through a whole lot of hoops (highly recommended) there is an amazon ami. Find the ami code by getting your amazon key, setting up ec2 command line tools and executing

thartman@ubuntu:~>ec2-describe-images -a | grep -i patchtag
IMAGE ami-febe5597 072945664613/patchtag dev
ami 072945664613 available public i386 machine aki-5f15f636 ari-d5709dbc ebs

You can run this ami from the comand line with ec2-run-instances, or use the aws gui or any of the numerous third party guis. Once you’re in, cd to the patch-tag directory, darcs pull the latest code, cabal install, and you should be good to go.

The main reason I am open sourcing patch-tag is to stimulate the haskell web-devel ecosystem by putting a “real world ready” app out into the open.

If anybody playing with the opened patch-tag wants to help me out with the project, here are my highest immediate priorities.

* Help me choose an open source license for patch-tag. I am considering gnu, bsd3, and CPAL (same license as reddit). What does the community think?

* Get patch-tag easier to install and on hackage. Patch-tag is not on hackage, because my feeling is that a program on hackage should Just Work or it violates legitimate expectations, and patch-tag cabal install is far from just working. Anybody is free to throw patch-tag on hackage under their own user account but I would say better to wait because the install issues *are* fixable with a little love and care.
Key subtodos for this are
* Unfork gitit and peg patch-tag to mainline gitit
* Use same ssh lib as darcsden so I can ditch most or all of the linux sysadmin stuff
* Make patch-tag machine instance for platforms other than ec2 (eg virtualbox, vmware, make it runnable on windows, etc).
* Diagnose and fix a suspected memory leak that results in regular visits from the dread oom killer.
* Transition patch-tag to a better understood / supported persistence solution than macid. My current thinking is sqlite, or possibly just text files with read/show serialization.
* Work on macid, and help realize the macid dream of straightforward nosql style persistence that uses native data structures for an easy start and scales to zillions of users with no unpleasant surprises.

Finally, and most importantly: Use haskell web-devel. Write documentation, ask questions, get on irc, and make the world a friendly place for people that like haskell and want to write the next facebook/ita/viaweb killer.

Thanks for tuning in, folks, and happy tagging!

Read Full Post »

patch-tag gets more ram

For free.

Thanks, linode!

Read Full Post »

Patch-tag Backups

I have gotten a few private queries about backups lately, and thought it would be good to say a few words about what and how we keep your repositories safe.

Patch-tag is hosted on linode, and we do backups using S3 in the amazon cloud.

Patch-tag backups up all repository and application state data once a day using backup manager. Archive files, which are in dar format, are split in 25 megabyte chunks before uploading to S3. Backups are full, not incremental, so every day is a complete snapshot of all repositories along with application (happpstack) state.The policy is to keep at least one month of backups.

After backups complete, I get a daily email stating the size of the data loaded into S3, along the lines of

20100312: backups look ok
repos: 9.24Gb
application state: 0.80Gb

In case of errors creating archive files or uploading, I get a scary email saying that something went wrong. (Haven’t gotten one yet though.)

As a sanity check, I have done two full recreations of patch-tag from backups, using the EC2 amis I described in an earlier blog post. [ link please] I am happy to report there were no unpleasant surprises.

As of now, backups are sitewide. So you can’t access your personal deleted repos, but in case of emergency you could make a request for this data, assuming it is less than a month ago. We do have plans to change this in the near future to provide you more control over the backups of you repos.

If there are additional backup features you would like to see, please let me know.

Happy tagging!

Read Full Post »

Mirroring Patch Tag

Gwern, darcs archivist extraordinaire, asked me to make it easier for him to mirror repositories hosted on patch-tag.

After some back and forth, I provided the link above.

If anybody else would like to mirror patch tag repositories, gwern kindly provided me with the script he uses for mirroring patch tag:

It’s a nice basic example use of tag soup for parsing simple html (a list of links), incidentally.

Enjoy!

Read Full Post »

Amazon EC2 is great, but it can get expensive, especially when you have an army of contractors doing all sorts of different things.

If you’re not careful, you can wind up creeping into a sort of tragedy of the commons situation where nobody shuts down little used EC2 instances because “that’s somebody else’s job” or simply because they’re not sure if anyone is using it. And since amazon bill doesn’t make it easy to see which server did what, let alone who was mostly using which server, it’s hard to formulate a good strategy for beating back the costs.

Here’s a simple, though somewhat radical solution. Prepay cloud developers for EC2 costs and have them use their own amazon accounts. The new improved amazon panel makes it easy to snapshot AMIs, do EBS volume management, and share these resources privately on a per-user basis. Share what you need to share, paypal the developer some funds and let them get on with it.

Suddenly your army of contractors is keeping a much more watchful eye on that EC2 balance because they don’t want to go back to their boss / client to request another prepaid-EC2 cash infusion more often than absolutely necessary.

This isn’t a silver bullet. If you are a fortune 500 company using “enterprise” cloud, you probably don’t want to start encouraging your sysadmins to start expensing basic server costs to their corporate credit card (actually, why not?…. nah). The prepay-EC2 strategy works best where you have a small well defined task that requires an EC2 environment, for example because setup is complicated and/or unautomated, and the developer is a cloudworker type outsourced contractor that wants to get coding with a minimum of friction. I think this approach could also work in a smallish startup environment casual enough that there is a degree of trust but using enouch computing resources that cost is beginning to matter. Prepaid EC2 might be just the trick for keeping the individual team members mindful of costs. I haven’t actually tried this though.

Minor detail: what do you do when the developer does such a good job minding costs that he completes the task far under the prepaid budget? I say, let them keep the excess!

Nota bene: the patch-tag production server is still in linode, and there are no concrete plans to switch to the cloud, though I am kicking the idea around.

Read Full Post »

The Happstack guestbook example application is the best way to learn how to learn HSP with happstack.

It is really cool, and HSP, haskell’s answer to php, has gotten really significantly better in the past few months.

The only thing is that it is ridiculously hard to find.

Hopefully that link three paragraphs back will help.

Read Full Post »

The Witching Hour at Starbucks

There is something eerie about the starbucks where I sit programming, around 7 pm. It closes at 9 pm, yet it feels closed already. Employees push mops around. The temperature drops. The light seems a little dimmer.

It is the witching hour.

Out of nowhere, the six or seven people left here start singing happy birthday to Berry, who has made us all many an espresso. We’re on pitch. Somebody is harmonizing, and we actually sound pretty good! Birthday girl is genuinely touched.

It’s a little warmer in here.

Read Full Post »

Patch-Tag uses linode, but I’ve decided to try out EC2 for development purposes, as the hourly pricing model is attractive for server instances that don’t need to persist, and… well, I’m curious.

First off, with a big bow of respect to the amz engineers, I’ll tell ya, EC2 has gotten a lot better lately.

The last time I used aws was 3 or so months ago when I was configuring cloud stuff for a video startup that used a lot of disk, in conjunction with rightscale management console. Everything worked from a technology standpoint, but my gut feeling vis a vis usability was that aws was an awful mess: guis that didn’t work right, command line interfaces with hard to find documentation, inconsistent flags… beh.

https://console.aws.amazon.com

is hugely improved, documentation and help seems to be better organized… everything is just just more polished… better.

Anyways, to get to the meat of my post, I have created two public amis that I hope may prove useful to developers and newbies alike.

EC2 Amazon Machine Image: ami-3122ce58

AMI ID: ami-3122ce58
Name: Haskell on EC2
Description: Based on elastic karmic, with haskell platform
Source: 072945664613/Haskell on EC2

and… a few cabal installs later

EC2 Amazon Machine Image: ami-3d23cf54

AMI ID: ami-3d23cf54
Name: Gitit Happstack Demo on EC2
Description: Gitit/Hapstack on EC2
Source: 072945664613/Gitit Happstack Demo on EC2

Now, these images are really not such a bit deal since what with cabal and the haskell platform, there have been huge gains in package installability. But they weren’t completely trivial either, due to missing dependencies, the quickcheck 1 to quickcheck 2 switch that broke a lot of happstack, and various other hoops I had to jump through…

*I* had to jump through.

*You* just load the AMIs.

I recommend to start here if you’re an aws newbie.

Happy hacking!

PS Check out the latest from Simon Marlow for another haskell on EC2 post


In case there any aws experts reading this, here is also a snip of a chat of me fumbling around. Feel free to comment or add your questions.


<patch-tag> I am creating an EBS bootable AMI, which takes (at least) > 10
minutes. Say five minutes after start I do “touch somefile”. Will
“somefile” be included in the new AMI? What governs this behavior?
<patch-tag> (I actually am performing this experiment at the moment but I am
curious as to what to expect and why)
<flaccid> um i havnt played with ebs boot yet, but i would assume so, its
meant to be persistent
<flaccid> try it out
<patch-tag> The reason I am asking is, say I want to snapshot while
configuring something, and I am at a safe/happy place. If the
snapshot applies to that exact moment in time, I can keep doing
dangerous things. But if the snapshot extends in time I have to
sit on my hands till it completes before continuing work.
<flaccid> well a snapshot is a snapshot
<patch-tag> er… ?
* flaccid looks up ebs boot
<patch-tag> not particular to ebs boot. question applies to any way of
creating amis.
<patch-tag> (sorry for obfuscating the issue)
<flaccid> oh right
<flaccid> well an AMI is indeed a snapshot
<flaccid> thus why it is called an image
<flaccid> looking at ebs boot, it does seem to be only the ability to boot
from ebs snapshots, so its not a persistent image but a manual way
of gaining persistence
<flaccid> and it costs more
<patch-tag> changing subject — costs more? I thought ebs storage when offline
was same as S3, 10c/G/month.
<patch-tag> also, AMI creation completed, so I’m actually going to check if
that file I created five minutes in got bundled…
<patch-tag> aaaaaand… it’s not there.
<patch-tag> I guess what’s getting snapshotted is ram in a certain state, not
hard disk. which is why it can be truly instantaneous.
<flaccid> everyone has said that

http://aws.amazon.com/about-aws/whats-new/2009/12/03/amazon-ec2-instances-now-can-boot-from-amazon-ebs/

works out costing more
<flaccid> patch-tag: where was the file on the filesystem you touched ?
<patch-tag> /home/ubuntu/testfile
<flaccid> afaik by default /home is not bundled
<flaccid> you will need to check that
<flaccid> and obviously ensure /home is actually on the root fs (/)
<patch-tag> no, it’s bundled. I know this from a previous snapshot.
<patch-tag> any insight on why boot from ebs costs more? I guess because if
you make a lot of snapshots those 10c/G/month add up?
<patch-tag> And the convenience of it makes it tempting to do more snapshots?
<flaccid> well if /home is bundled then, i have no idea. bundling is
bundling…
<flaccid> and yeah can’t answer your second question. unfortunately i’m not
amazon sorry
<patch-tag> just in case it’s not clear, we’re in good shape. this is the
behavior I want.
<patch-tag> My assumption is that snapshot is point in time, and I can
initiate dangerous configs after initiating snapshot without
fear.
<flaccid> like i was saying before, yes a snapshot is a snapshot
<flaccid> whether it is a bundled ami or ebs snapshot, they are both snapshots

Read Full Post »

Older Posts »

Follow

Get every new post delivered to your Inbox.