Persistence Thoughts

Necessity is the mother of invention. I've been playing around with ideas for a persistence layer for a while now, but it wasn't until I got a hard deadline for a database-driven website that things really clicked. I want to share these thoughts here, both to get them down in writing, and to hopefully provoke some discussion.

There's some version of the code I discuss here available on github. It doesn't match the code here precisely, and is not what the final version will look like. But it does show how we can support multiple backends!

Pieces of the puzzle

There are going to be three separate components of persistent:

• A high-level interface for declaring entities. There will be a set of datatypes, and perhaps either a DSL, Quasi-Quoted syntax or external files for a nicer user interface.
• A set of type-classes for doing the actual operations on tables. The majority of this post will discuss this.
• Each backend will have its own Template Haskell code for generating instances of the type classes based on the higher-level interface. The SQL generators will probably share a lot of code, but that's mostly irrelevant to the outside user.

Type families

My proposal makes heavy use of type families. If you want a quick introduction to type families, they're basically a method to associate two different datatypes together.

Let's give a pertinent example. Every record in a table will have a unique key associated with it. If we're talking about a SQL backend, this will most likely be an auto_increment (MySQL) or serial (PostgreSQL) integer, but Amazon SimpleDB prefers to use UUIDs. So I want to associate a key value based on the datatype of the value and the backend. In type families, this might look like:

data family Key value backend

However, in general we'll be defining these inside typeclasses, so it will look more like:

class HasTable value backend where
    data Key value backend

Then you would declare an instance like so:

instance HasTable MyValue MyBackend where
    data Key MyValue MyBackend = MyKey Integer

No relation

I've mentioned a number of times in previous posts the idea of no relations. Let me clarify here: persistent will not automatically load related data from other tables. There are three good reasons for this approach:

1. Not all backends natively support relations. Think of a set of CSV files, or Google BigTable for that matter.
2. Relations will greatly complicate things, and I really want something simple.
3. Arguably, there's a performance edge. But I really don't want to discuss performance right now.

Multiple type classes

So without relations, every record is really just a single row in a single table. We need to be able to do two things:

1. Convert our value to a record.
2. Convert a record to our value.

That's all well and good, but what's a record? If we're talking about CSV files, it's [String]. But in a database (using HDBC) it's [SqlValue].

So we'll need to have different typeclasses for different backends. However, since we have multiple Template Haskell generators, all is fine: the CSV TH code knows to use the IsStringList class, while the PostgreSQL TH uses IsSqlValueList... or whatever it's called.

However, some database operations require more than just this. What if you want to update a record? What if you want to search? Or sort?

I propose a set of classes akin to the following:

class HasField a where
    data Field a
    applyField :: Field a -> a -> a

class HasFilter a where
    data Filter a
    applyFilter :: Filter a -> a -> Bool

class HasOrder a where
    data Order a
    applyOrder :: Order a -> a -> a -> Ordering

Let's see a sample set of instances.

data Person = Person String Int
instance HasField Person where
    data Field Person = PersonName String | PersonAge Int -- type safe!
    applyField (PersonName name) (Person _ age) = Person name age
    applyField (PersonAge age) (Person name _)  = Person name age

instance HasFilter Person where
    data Filter Person = PersonNameEq String -- name equality
                       | PersonAgeLt Int     -- age is less than parameter
    -- we could add as many filters here as we like
    applyFilter (PersonNameEq x) (Person y _) = x == y
    applyFilter (PersonAgeLt x) (Person _ y) = y < x="" instance="" HasOrder="" a="" where="" data="" Order="" Person="NameAsc" --="" name,="" ascending="" |="" AgeDesc="" --="" age,="" descending="" applyOrder="" NameAsc="" (Person="" x="" _)="" (Person="" y="" _)="compare" x="" y="" applyOrder="" AgeDesc="" (Person="" _="" x)="" (Person="" _="" y)="compare" y="" x="" --="" reverse<="" /code="">

class HasField v => Updateable v backend where
    update :: Key v backend -> [Field v] -> IO ()
class HasFilter v => Searchable v backend where
    select :: [Filter v] -> IO [(Key v backend, v)]

class HasFilter v => SqlFilter v where
    toWhereClause :: Filter v -> String
    toWhereData :: Filter v -> SqlValue
instance SqlFilter Person where
    toWhereClause (PersonNameEq _) = "name=?"
    toWhereClause (PersonAgeLt _) = "age<?"
    toWhereData (PersonNameEq x) = SqlString x
    toWhereData (PersonAgeLt x) = SqlInteger x