MongoDB, replica sets and authentication

I've just recently had to set up a MongoDB replica set using Chef at work. It was also required that authentication be set up in MongoDB.

I had very little experience of either technology (but I would now recommend both) so it took me quite a while to figure it all out so I thought I'd offer some advice here for anyone else who may need it. Although I used Chef most of this stuff still applies even if you're doing it manually. I'll be presuming you know what replica sets are, if you don't but are interested please read up on them first.

Many set-ups are possible with replica sets but we opted for a three node set: primary, secondary and an arbiter.
It's also worth noting that I used Vagrant to test all this which I'd highly recommend. It did mean having three VM's running simultaneously which chewed up my RAM but it was worth it.

We used the edelight chef-mongodb cookbook as the starting point. I added recipe[mongodb::replicaset] to all three nodes' run lists. Then came my first stumbling block: by default this recipe will try to initiate the replica set from all three nodes but only the primary can do that so, for the other two nodes, you need to add the following attribute:

{
    "mongodb": {
        "auto_configure": {
            "replicaset": false
        }
    }
}

Then on all three nodes you need to add the following (this can be combined with the other mongodb attributes, it's just shown separately here for convenience):

{
    "mongodb": {
        "cluster_name": "ClusterName", 
        "replicaset_name": "ReplicasetName"
    }
}

And finally on the arbiter node you need to add:

{
    "mongodb": {
        "arbiter": true
    }
}

The next problem was that, at the time of writing, this cookbook does not support authentication. There is a pull request to add keyFile support (which is how you do authentication for replica sets) but it has not been actioned yet. We forked the repository and pulled in these changes ourselves. Once done you then need to generate suitable keyFile contents then specify the following attribute on all three nodes:

{
    "mongodb": {
        "key_file": "KeyFileContentGoesHere"
    }
}

Non-Chef note:
If you're doing this manually you'll need to make sure you start mongo with the following arguments (Chef does this for you):

--replSet ReplicasetName --keyFile /path/to/keyFile

The next issue was that this cookbook has been written for Chef-Server and tries to search for the nodes of the replica set to initiate. Chef-Solo, however, does not support searching. Edelight offer a workaround for this with chef-solo-search. I didn't actually make use of this, we just hard-coded our list of nodes into some attributes but that's a bit hacky.

On to the next problem: the set members contact each other via their domain names. Not all of our servers had domain names set up so we needed to add hosts entries to each server in the set. In Chef we did this with CustomInk's Hostsfile cookbook.

The sequence in which you bring up the replica set nodes is somewhat important in that they must all be operational before you can initiate the set, therefore the primary has to come up last.
I provisioned the arbiter first (which we were already using for something else) and ensured that mongodb was running and that I hadn't broken anything else. I then provisioned our secondary and checked that mongo was running and that it could ping the domain name of the arbiter (which was in the hosts file) and vice-versa. I then provisioned the primary which initiated the replica set. After about a minute it was up and running.

Well... there was a little more to it than that.
We already had the 'primary' MongoDB server live but just as a standalone, we'd later decided to make it a replica set. We'd already set up authentication on the standalone and that's where things got complicated. In order to initiate a replica set on a MongoDB instance with authentication you must be authenticated as a user with the clusterAdmin role. If you don't you will simply get an “unauthenticated” error which, if you're like me, will make you tear your hair out as you're sure you're entering the right credentials.
The other issue is that the edelight cookbook does not handle authenticating before trying to initiate the replica set so we had to add this in. It's very simple, in libraries/mongodb.rb, in the configure_replicaset define, just before the command to initiate the replica set add:

admin.authenticate("username", "password")

As long as that user has the aforementioned clusterAdmin role you should be set.