Marten & ASP.NET Core - Postgresql noSQL storage in ASP.NET Core

What is Marten and why do I care?

Marten (no relation) is an open source .NET driver for Postgresql that focuses on Postgres’ JSON-based document storage capabilities. Think of it like an alternative to MongoDB or RavenDB for your .NET applications.

Getting started

This article is going to walk you through the first steps of getting an ASP.NET Core project up and running with Marten, as well as basic querying. This should work on all operating systems.

The first thing you’ll want to do is start a new ASP.NET Core project. Remember, in the new .NET Core world an ASP.NET Core project is just a regular old .NET Core console app that hosts its own web server and uses ASP.NET libraries. This guide is the canonical reference for getting started here. You’ll end up with three files in your project that look like this.

using Microsoft.AspNetCore.Hosting;

namespace Sample
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var host = new WebHostBuilder()
                .UseKestrel()
                .UseStartup<Startup>()
                .Build();

            host.Run();
        }
    }
}
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;

namespace Sample
{
    public class Startup
    {
        public void Configure(IApplicationBuilder app)
        {
            app.Run(context =>
            {
                return context.Response.WriteAsync("Hello from ASP.NET Core!");
            });
        }
    }
}
{
  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable",
    "emitEntryPoint": true
  },
  "dependencies": {},
  "frameworks": {
    "netcoreapp1.0": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        },
        "Microsoft.AspNetCore.Server.Kestrel": "1.0.0"
      },
      "imports": "dnxcore50"
    }
  }
}

Add a model and a controller

Let’s quickly add a model - this will be the object that maps to our table in Postgresql. Create a new Model directory and add a BlogPost model.

using System;
using System.Collections.Generic;

namespace Sample.Model
{
    public class BlogPost
    {
        public int Id { get; set; }
        public string Title { get; set; }
        public string Body { get; set; }
        public IEnumerable<string> Tags { get; set; }
        public Dates ModifiedDates { get; set; }

        public class Dates 
        {
            public DateTime CreatedDate { get; set; }
            public DateTime LastEditedDate { get; set; }
        }
    }
}

And we’ll need a controller which manages this. Create a new Controllers folder and create a file called BlogPostController.cs.

using System.Collections.Generic;
using Microsoft.AspNetCore.Mvc;
using Sample.Model;

namespace Sample.Controllers
{
    [Route("/posts")]
    public class BlogPostController
    {
        [HttpGet]
        public IEnumerable<BlogPost> Get()
        {
            // TODO get all blog posts from DB
            return new BlogPost[0];
        }

        [HttpGet("{id}")]
        public BlogPost Get(int id) 
        {
            // TODO get the specified blog post from DB
            return null;
        } 

        [HttpPost]
        public BlogPost Create([FromBody]BlogPost post)
        {
            // TODO create a blog post in the DB
            return null;
        }
    }
}

For this step you’ll also need to add a depedency on Microsoft.AspNetCore.Mvc to project.json - while we’re here, add dependencies on Kestrel (the built-in web server for aspnetcore development) and Marten:

{
  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable",
    "emitEntryPoint": true
  },
  "dependencies": {
    "Microsoft.AspNetCore.Server.Kestrel": "1.0.0",
    "Microsoft.AspNetCore.Mvc": "1.0.0",
    "Marten": "0.9.12.563"
  },
  "frameworks": {
// ...etc

Finally before using this controller you’ll need to setup the ASP.NET MVC framework. Edit your Startup.cs file:

using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Marten;
using Microsoft.AspNetCore.Hosting;

namespace Sample
{
    public class Startup
    {
        public void Configure(IApplicationBuilder app)
        {
            app.UseMvc();
        }

        public void ConfigureServices(IServiceCollection services)
        {
            // Add framework services.
            services.AddMvc();
        }
    }
}

At this stage it’s a good idea to fire up the web server and make sure everything is working as expected. Go to your command line terminal and run ‘dotnet run’. It should tell you that the server is running on localhost on some port (default is 5000). Open up a browser, Postman or your HTTP client of choice and make sure that when you hit http://localhost:5000/posts an empty array is returned. If not, something has gone wrong.

Configure dependency injection

The next step is to get the Marten DocumentStore object injected into our controllers so we can use it. Open up your Startup.cs file and edit the ConfigureServices method so that it looks like below. ConfigureServices is the method that the ASP.NET Core framework will call to set up your dependency injection framework - a simple DI container is included with ASP.NET core, so that’s what we’re going to use.

public void ConfigureServices(IServiceCollection services)
{
    // Add framework services.
    services.AddMvc();

    // Marten document store
    services.AddScoped<IDocumentStore>(provider => 
        DocumentStore.For("Server=127.0.0.1;Port=5432;Database=my-database;User Id=admin;Password=admin;"));
}

This line tells the dependency injection framework that we want to be able to ask for instances of IDocumentStore, and when we ask for them you should call that DocumentStore.For() method to get one. Additionally because we use the AddScoped method, this object will exist for the lifetime of a single request. You can read more about object lifetimes in ASP.NET Core here.

Obviously you’ll need to change your connection string to point to your own Postgresql instance. You can find more information about Postgresql connection strings here.

Querying in the controller

Now we should be able to just add an IDocumentStore to the controller’s constructor and the framework will supply it to us, ready to use. Let’s give it a crack. First add a constructor to the controller which takes the IDocumenStore and stores it in an instance variable.

private readonly IDocumentStore _documentStore;

public BlogPostController(IDocumentStore documentStore)
{
    _documentStore = documentStore;
}

And let’s flesh out the methods we defined earlier.

[HttpGet]
public IEnumerable<BlogPost> Get()
{
    using (var session = _documentStore.QuerySession())
    {
        return session.Query<BlogPost>();
    }
}

[HttpGet("{id}")]
public BlogPost Get(int id) 
{
    using (var session = _documentStore.QuerySession())
    {
        return session
            .Query<BlogPost>()
            .Where(post => post.Id == id)
            .FirstOrDefault();
        
    }
} 

[HttpPost]
public BlogPost Create([FromBody]BlogPost post)
{
    using (var session = _documentStore.LightweightSession())
    {
        session.Store(post);
        session.SaveChanges();
        return post;
    }
}

Let’s dig into what we’re doing here. The IDocumentStore can supply sessions which we use to query the database.

You may notice we’re using two different methods to get the DB session - QuerySession() and LightweightSession(). The query session is a read-only session, optimised for read scenarios. LightweightSession is a read-write session which requires you to manage object change tracking yourself.

The other alternatives are OpenSession() which supplies an implementation of IDocumentSession which is backed by an identity map - basically an in-memory cache of any data that has been retrieved. And finally there’s DirtyTrackedSession() which is a session which will track changes you make to any retrieved entities by storing the original document in memory. I don’t recommend using this unless you really need to - other than the performance penalty, in my experience it’s simpler to be explicit with your changes.

Once we have the session we can query it by calling Query<T> where T is the document type we’re trying to look up. The object returned by Query() is a Linq IQueryable so you can call most common Linq extension methods on it (although you should be cautious with how you query a document database - see more query information here).

Test it out

To test this, fire up your app an insert a document by issuing a POST request to http://localhost:5000/posts that looks something like:

{
    title:         "My new blog post",
    body:          "<h2>a blog post</h2><p>this is my new blog post</p>",
    tags:          ["marten", "blog"],
    modifiedDates: {
        createdDate:    "08/09/2015 21:00",
        LastEditedDate: "08/09/2015 22:50"
    }
}

And you’ll receive a response:

{
  "id": 1001,
  "title": "My new blog post",
  "body": "<h2>a blog post</h2><p>this is my new blog post</p>",
  "tags": [
    "marten",
    "blog"
  ],
  "modifiedDates": {
    "createdDate": "2015-08-09T21:00:00",
    "lastEditedDate": "2015-08-09T22:50:00"
  }
}

Interestingly you’ll find that Marten has automatically assigned a value to the Id field on our object. When you use an integer or long, Marten will use an auto-incrementing algorithm, and it will generate GUIDs if your Id field is a GUID. See here for more info on Marten object identity.

You can now point your web browser at http://localhost:5000/posts to see a listing of every document you have inserted, or http://localhost:5000/posts/{id} to find the document with that ID.

Wrapping up

The most interesting thing is how Marten is using Postgres to store your data. If you open your database instance you’ll find a table that’s defined something like

CREATE TABLE public.mt_doc_blogpost
(
  id integer NOT NULL,
  data jsonb NOT NULL,
  mt_last_modified timestamp with time zone DEFAULT transaction_timestamp(),
  mt_version uuid NOT NULL DEFAULT (md5(((random())::text || (clock_timestamp())::text)))::uuid,
  mt_dotnet_type character varying,
  CONSTRAINT pk_mt_doc_blogpost PRIMARY KEY (id)
)
WITH (
  OIDS=FALSE
);
ALTER TABLE public.mt_doc_blogpost
  OWNER TO admin;

The interesting bit is “data jsonb NOT NULL” which is the column that stores all of the data. The json document stored in there looks just like the one that was returned after your POST request. The jsonb data type in Postgresql is much more than just a JSON string - there’s real potential to do some cool stuff with this data type, and that’s why Marten is so interesting.

Read more about Marten here, and check out the code. You can find the full source code for this guide at my Bitbucket.

Prefix tree (trie) in F#

What is a prefix tree?

A prefix tree - otherwise known as a trie (pronounced “tray”) - is a data structure designed for fast searching over prefixes - for example given the list of words [CAT, CATS, CART, CARTS], which words are prefixed with “CAR”? Performing this search on a hashset would require you to check every value (O(n)) complexity). A sorted list would need to perform a fast search for the first item with the prefix and then scan or search the list until no matching items remain - O(logn) + O(logm) where m is the sublist which has the prefix - which is fast, but a sorted list is slow to build, slower than a hashset to search for single items and slow to tell me if a given string is a valid prefix to any word in the list.

With a prefix tree this search is performed in O(m*p) time (again m is the sublist of prefixed strings, and p is the length of the average result string - typically very small compared to the string list) and search for a single string in the prefix tree is performed in O(p) where p is the length of the string to search for (not as fast a a hash set, but very near for most cases). At a glance this seems worse than a sorted list, but in typical use cases, such as storing dictionary words, the length of the word you’re searching for is a much smaller number than the number of words you’re storing.

And the coolest thing about a prefix tree is that it’s actually really simple. A prefix tree is a node with (optionally) a value, and a map of values to subnodes. To construct a prefix tree, start with a node with no value and add a string by appending that node’s map with the first character, mapping to a new trie with that character as its value, and add the rest of the string to that node recursively in the same way.

Implementation

I have implemented a simple prefix tree in F#. The code is available here and you can find it on nuget. This trie works well in C# and F# (and any other CLR language) and unlike other F# prefix trees I’ve spotted around the internet, mine is completely immutable.

Here’s an example of usage from F#:

open FTrie
[<EntryPoint>]
let main argv = 
    let t = Trie(["abcde";"abba";"ababab"])
    printfn "%A" (t.getWords()) // seq ["ababab"; "abba"; "abcde"]

    let t2 = t.withWord("cdfge");
    printfn "%A" (t2.getWordsPrefixedBy("ab")) // seq ["ababab"; "abba"; "abcde"]
    0

And in C#

using FTrie;
using System;
class Program
{
    static void Main(string[] args)
    {
        Trie t = new Trie(new[] { "cars", "carts", "cats", "cartoons" });
        Console.WriteLine(string.Join(", ", t.getWordsPrefixedBy("car"))); // cars, cartoons, carts

        var t2 = t.withWord("carve");
        Console.WriteLine(t.isPrefix("car")); // True
        Console.WriteLine(t.isPrefix("cra")); // False;
        Console.ReadLine();
    }
}

A better add function

This immutable implementation creates a new trie when you call the withWord() function instead of adding the word to the existing data structure. It does this by just pulling out every existing word and calling the Trie constructor with the plus the new one. With very large tries this could become quite memory and computationally intensive.

You could implement a better add function using the following algorithm. I didn’t bother because I didn’t need this functionality, so I leave it as an exercise for you (feel free to open a pull request into my repo if you find a good solution).

add(word)
1. currentNode = root
2. while currentNode contains first letter of word
3.   currentNode = currentNode.children[first letter of word]
4. clone currentNode except with a new children map containing the rest of this word chain
5. replace the chain of parents with new tries that reference the new nodes

What you’ll end up with is two trees that reference identical nodes in all but one branch and have different roots. This would be much faster than creating whole new prefix trees from scratch, and use less memory. Because all branches are immutable no one with a reference to another trie could mess around with data in your branches. Don’t forget to account for the EOW bit!

New scriptcs script pack - MemberPrint

I have just published a really simple little ScriptCs (I'm still not sure if I should be capitalising that or not) script pack. It's intended to be used on the REPL. All it does it print some details about the objects and types to help you navigate the REPL more easily. As usual, the code is available on GitHub.

To install, run the following command:

scriptcs -install scriptcs.memberprint

Then just run scriptcs. The easiest way to figure out the API is to run it over itself, like this:

c:\test>scriptcs
scriptcs (ctrl-c or blank to exit)
> var print = Require();
> print.Methods(print);
+   Constructors(Type t, BindingFlags flags, String regex, Boolean verbose) : Void
+   Constructors(Object o, Boolean verbose) : Void
+   Constructors(Object o, String regex, Boolean verbose) : Void
+   Constructors(Type type, Boolean verbose) : Void
+   Constructors(Type type, String regex, Boolean verbose) : Void
+   Constructors(Object o, BindingFlags flags, Boolean verbose) : Void
+   Constructors(Type t, BindingFlags flags, Boolean verbose) : Void
+   Constructors(Object o, BindingFlags flags, String regex, Boolean verbose) : Void
+   Equals(Object obj) : Boolean
+   Events(Object o, Boolean verbose) : Void
+   Events(Object o, String regex, Boolean verbose) : Void
+   Events(Type t, Boolean verbose) : Void
+   Events(Type t, String regex, Boolean verbose) : Void
..... (etc)

MemberPrint supports a few filtering options. You can filter using BindingFlags, just as though you were doing the reflection yourself:

> print.Methods(new System.Text.RegularExpressions.Regex(""), BindingFlags.Static|BindingFlags.Public);
+ # CompileToAssembly(RegexCompilationInfo[] regexinfos, AssemblyName assemblyname) : Void
+ # CompileToAssembly(RegexCompilationInfo[] regexinfos, AssemblyName assemblyname, CustomAttributeBuilder[] attributes) : Void
+ # CompileToAssembly(RegexCompilationInfo[] regexinfos, AssemblyName assemblyname, CustomAttributeBuilder[] attributes, String resourceFile) : Void
+ # Escape(String str) : String
+ # get_CacheSize() : Int32
+ # IsMatch(String input, String pattern) : Boolean
..... (etc)

You can also filter using regular expression text.

> print.Methods(new System.Text.RegularExpressions.Regex(""), "^Is.+");
+ # IsMatch(String input, String pattern) : Boolean
+ # IsMatch(String input, String pattern, RegexOptions options) : Boolean
+ # IsMatch(String input, String pattern, RegexOptions options, TimeSpan matchTimeout) : Boolean
+   IsMatch(String input) : Boolean
+   IsMatch(String input, Int32 startat) : Boolean

You can run it over types as well as instances. Can't remember the constructors for System.DateTime?

> print.Constructors(typeof(DateTime))
+   .ctor(Int64 ticks)
+   .ctor(Int64 ticks, DateTimeKind kind)
+   .ctor(Int32 year, Int32 month, Int32 day)
+   .ctor(Int32 year, Int32 month, Int32 day, Calendar calendar)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second, DateTimeKind kind)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second, Calendar calendar)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second, Int32 millisecond)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second, Int32 millisecond, DateTimeKind kind)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second, Int32 millisecond, Calendar calendar)
+   .ctor(Int32 year, Int32 month, Int32 day, Int32 hour, Int32 minute, Int32 second, Int32 millisecond, Calendar calendar, DateTimeKind kind)

There are multiple overloads for each of the following methods:

  • Methods(object o)
  • Properties(object o)
  • Events(object o)
  • Constructors(object o)
  • Members(object o) (this one just calls all of the others)

The code is a little rough around the edges at the moment, but so far I have found it useful enough that I decided to share the script pack.

Solutions to some problems hosting Windows Workflow Foundation applications in AppFabric

I'm trying to stand up some Windows Workflow Foundation applications in AppFabric for a spike on an application I'm building and so far it has been... unpleasant. Here are a couple of solutions to some frustrating problems I encountered. Hopefully I can help save someone else a bit of time.

Workflow persistence is not functional because the net.pipe binding is not enabled for this web site

Encountered while trying to configure my WF service on an IIS virtual directory. In particular I encountered this error on the Workflow Persistence tab and a similar one onthe Workflow Host Management tab in the Configure WCF and WF for Application dialog.

Enable the net.pipe protocolNow, in theory this will be automatically fixed when you click Apply. And in theory, even if it's not then it should be resolved by taking the following steps:

  • Right click the web site that hosts your service in the IIS Connections pane, click Manage Websites -> Advanced Settings and add ,net.pipe to the end of the "Enabled Protocols" setting (no spaces!)
  • With the same site selected, click Edit Bindings on the Actions pane and add a net.pipe binding with Binding Information = *

And in theory, practice and theory are the same. But in practice, unfortunately, they are not. If you're not running Windows Server (or your install is a bit wonky) then there's an extra step to take, because you'll find that net.pipe is not an available binding type in the bindings dialog.

To resolve: Open the "Turn Windows features on or off" dialog (find it by searching in the Start menu/screen, under Settings). Open up the Microsoft .NET 3.5 folder and tick "Windows Communication Foundation Non-HTTP Activation". This should make the net.pipe binding available to you. You may need to run iireset in a console as admin first.


Error: Unable to start debugging on the Web Server

or, if you try running without debugging

Error: Cannot obtain Metadata from xamlx The remote server returned an unexpected response: (405) Method Not Allowed.

or, if you drill further into that error

HTTP Error 404.3 - Not Found The page you are requesting cannot be served because of the extension configuration. If the page is a script, add a handler. If the file should be downloaded, add a MIME map

The honest truth is I'm not sure what caused this error. I tried a whole bunch of stuff. I added some handlers to my root Web.config, installed some Workflow hosting modules in IIS, played with permissions, all kinds of stuff and I couldn't get the site to handle the .xamlx request. Ultimately here's what resolved the issue for me. The first step may not be necessary, I honestly don't know (I recommend reading the linked article). But it worked.

  • Run the following command: "%WINDIR%\Microsoft.Net\Framework\v3.0\Windows Communication Foundation\ServiceModelReg.exe" -r
  • Uninstall and reinstall the following Windows features from the "Turn Windows features on or off" dialog: Windows Communication Foundation HTTP Activation, Windows Communication Non-HTTP Activation. They are both under the .NET Framework 3.5 feature.

applicationHost.config : Unrecognised attribute: 'serviceAutoStartMode'

After I got my workflow up and running I realised that every other service and website on my machine was broken. It seems that AppFabric stepped all over my applicationHost.config file and left an invalid attribute in there. Take a look at this thread for the solutions to this issue.

Building a scriptcs script pack

If you haven't seen scriptcs yet you should check it out. It's a scripting environment for C#, but more interestingly it incorporates a module system similar to what you'd see in node.js using NuGet as a packaging mechanism. I won't go into an intro to scriptcs, because others like Scott Hanselman and the man himself, Glenn Block (and contributors) have done a better job that I could have.

Instead I'll talk about a concept that (so far) hasn't had a lot of documentation love - script packs.

What is a scriptcs script pack?

The basic idea is that a script pack is an object (generally compiled in a DLL and available in your script's bin directory) that either exposes some functionality to the scripting environment or simplifies the usage of some other functionality. This seems like an overly broad definition so let's start with an example of script pack usage. From the scriptcs.WebApi script pack.

public class TestController : ApiController {
  public string Get() {
    return "Hello world!";
  }
}
var webApi = Require();
var server = webApi.CreateServer("http://localhost:8080");
server.OpenAsync().Wait();
Console.WriteLine("Listening...");
Console.ReadKey();
server.CloseAsync().Wait();

That's the complete script - no using statements and ridiculously simple API. The script pack takes care of "injecting" using statements into any script that uses the script pack (ie, calls that Require method above - we'll get into that) and returning a script-friendly object to the shell.

Let's get started building a script pack. I want to create a script pack that allows me to pull down arbitrary strings from an HTTP request - let's keep it really nice and simple.

Fire up Visual Studio (note: you don't actually need Visual Studio for this, but I'll assume if you're comfortable enough to use other tools then you'll know when to adjust the steps) and create a new Class Library project. Delete that damn Class1.cs file, open the Package Manager console and execute the command

Install-Package scriptcs.contracts

The contracts library has a couple of interfaces we'll need to implement to make our script pack work. Let's start by creating our "convenience object". We'll call it Http, and all it's going to do is make GET requests and return strings. So let's implement this class.

using System.Net;
using ScriptCs.Contracts;
namespace ScriptCs.Http
{
    public class Http : IScriptPackContext
    {
        public string Get(string uri)
        {
            var client = new WebClient();
            return client.DownloadString(uri);
        }
    }
}

That IScriptPackContext interface there is a marker interface that scriptcs seems to use to identify objects that are capable of being returned by script packs. It has no members, so there's nothing to implement.

The next step is to implement the script pack itself. This object will be picked up by the scriptcs engine and is capable of generating and returning IScriptPackContext objects. Because our script pack is so simple we don't need to do much in the implementation of this interface:

using ScriptCs.Contracts;
namespace ScriptCs.Http
{
    public class HttpScriptPack : IScriptPack
    {
        IScriptPackContext IScriptPack.GetContext()
        {
            return new Http();
        }
        void IScriptPack.Initialize(IScriptPackSession session)
        {}
        void IScriptPack.Terminate()
        {}
    }
}

This is fairly self-explanatory. The scriptcs engine will cache all available IScriptPack objects and when a particular script pack is requested (see below) then the appropriate script pack context will be returned.

The Initialize method here is interesting. We're not doing anything with it in this particular program, but it's worth going over what can be done. This method gives us an IScriptPackSession object which has two important methods:

  • AddReference - use this method to add library references to be available in your script. After your script pack is loaded into a script the specified references will be available for your use with no further code inside the script.
  • ImportNamespace - this method can import namespaces for use in your scripts. It's just a nice convenience so that your user's scripts can stay nice and clean. We don't need to do this in our current script pack, but if you refer back to the WebApi script above you'll notice we're using the ApiController class without any using statements. This is because the WebApi script pack imports this namespace for you.

Obviously the Terminate method should be used for cleaning up any resources after yourself and leaving the machine in a good state.

So that's pretty much all there is to building a script pack. Now we'll go ahead and use it. Build your library, grab the DLLs from your bin directory and move it into a test folder like c:\myscripts\bin. Create a new script that looks like

var http = Require();
Console.WriteLine(http.Get("http://google.com"));

Save it into your scripts folder (c:\myscripts\myscript.csx in my above example) and execute it using

scriptcs myscript.csx

And observe the output. All done! Now all that's left is to package your script up in a NuGet package and publish.