Adding a little varnish

Since the latest release of Yesod I've been working on consolidating Yesod information onto my new VPS system. I'd previously kept this blog on my personal domain name and the docs site separate, and generated both of them as static sites (the former via a custom script, the latter via Hakyll).

However, I decided to combine both sites into a single Yesod site; you are reading the result. In the process, I've put the majority of the old documentation into the book and created separate examples and screencasts sections. Hopefully, this should make it much easier to find information.

Testing examples

All of the examples on the site double as executables in the yesod-examples package on hackage. This provides a simple means to test that the examples are written correctly and still compile with new versions of Yesod. However, up until recently, examples in the rest of the documentation and the book had no such testing involved.

So I devised a very simple solution: there's a snippets folder that contains various Haskell files, each one a stand-alone executable. The book simply embeds pieces from those files for the code snippets. Also, I finally added in syntax highlighting via hscolour.

Varnish

Anyway, the part that I thought people would actually be interested in: loading up all of these various static files for each request and performing all the syntax highlighting and markdown rendering is a fairly heavy performance price. At first, I'd thought about playing with caching the results in temporary files and checking file modification times.

Instead, a much simpler solution presented itself: Varnish, an HTTP accelerator that performs caching. I'd read about it a while ago, but never found it necessary, since most of my Yesod sites are purely dynamic sites, which are not amenable to caching.

Setting up Varnish was incredibly simple: I already run my nginx server on port 8080 and have port forwarding redirect traffic from port 80. Now, Varnish simply listens on port 8081 and port forwarding redirects 80 to 8081. And I get to have that warm, fuzzy feeling of using some new technology I read about on Reddit. Yeah!

The only code change required to take advantage of Varnish is an HTTP cache-control response header, which is already easy enough with the setHeader Yesod function. I'll probably make a minor release of Yesod (0.5.1?) to include some helper functions to make this easier.

Which reminds me...

It's a real pain getting a VPS set up. Now that I'm hosting a number of different Yesod apps from my VPS now, it's fairly easy and straightforward to add more Yesod apps to it. Each Yesod app requires a very small amount of resources (< 20M of RAM and not too much CPU), but still requires a VPS with enough RAM to actually pull off the compiles itself.

This makes me think Yesod apps are quite amenable to some kind of managed hosting. I've thought about this a little bit, and here's the idea I've come up with:

• We would need some kind of standard format for applications. I think a cabalized tarball is the perfect fit.
• Have a requirement that each cabal file have a rule to build an executable called fastcgi, and build this file with the cabal flag production. This is what the current scaffolder does. (A future idea would be to design a file that could use any WAI backend, but I wanted to start simple.)
• Since Template Haskell code can be run at compile time, for security the build of the process should occur in a chroot jail.
• Running the application would also occur in a chroot jail. All of the source code from the tarball would be present in the chroot jail, so extra resources (settings files, etc) would be present. Static files should follow the standard layout: in the static folder.
• As per my deploying advice, use spawn-fcgi and bind to a named socket, and have an nginx config block automatically generated that talks to that socket. Perhaps some information on domain name should be included in the tarball, of perhaps that would be part of a web-based admin util.

I'm kind of tempted to set up an automated system like this, if only to automate the management of my current sites. If anyone would be interested in this kind of hosting, and would like to help me test this out, let me know. I still have a little bit of extra room on my VPS to play around with.