Kingboard & TokuMX for MongoDB

EDIT: TokuMX v1.3.0 brought full MongoDB v2.4, so its not based on v2.2 anymore like claimed in this post.

When i started with Kingboard it was mostly to try out MongoDB, and to learn more on how NoSQL Databases could be used. A year later EVSCO started sponsoring the project – ever since it has slowly been growing. The Virtual Server that Kingboard.net is running on has limited hard disk space. Lately Karbowiak has helped me to push about 8million api confirmed killmails into the MongoDB that it is running on. One of the results being that the harddisk usage of the Database (Killdata + EVE data + indexes) grew to above 60GB. MongoDB itself has been quite useful & stable for the killboard, but since it stores the documents uncompressed it actually wastes *a lot* of space.

Karbowiak and me have been talking about using TokuMX for a while now. TokuMX is a MongoDB fork by tokutek (they also have tokudb, a storage engine for mysql), which uses fractal tree indexes, compression and enables transactional safety for MongoDB, all in all, if you read their page (linked above) it reads to good to be true. Saving space + better performance + better features? Who wouldn’t love that.

There are a few obvious downsides to TokuMX, like very low adoption, an extreme small community and virtually no documentation / blogs on it (Besides a few guys celebrating its creation). Then there are some downsides that i will mention during this blogpost, which i found when actually fiddeling with it. One thing to note is that TokuMX is based on a 2.2 mongo version, and had some 2.4 features backported.

So i started to convert my MongoDB installation to TokuDB, basically the first step was Karbowiak adding another virtual harddisk, so i had enough space to make a mongodump of the database. Then i downloaded a tarball with the TokuMX binaries (didn’t feel like building it from source) and run the mongorestore that came with TokuMX (thats important, since TokuMX and MongoDB are not binary compatible). The import itself went quite quick, but building the indexes took several hours.

Once i was done with the import i started up the daemon, booted up the webserver and visited kingboard – or at least tried to. When the side hadn’t loaded after two minutes i started digging through logfiles. With MongoDB the “kills” side had been one of the fastest to load, and i didn’t see any reason why that should be different. After two more minutes the reason popped up in tokumx’s logfile, the count() done to get the total count of kills took over 4 minutes!

I ran to the IRC Channel of tokutec, hoping to get some help there (maybe i did something wrong). As i mentioned the community is extremly small, so i was out of luck, no one was awake to reply. So i fired up my browser, googled and started reading some of the sourcecode of TokuMX. Apparently MongoDB does a count() by answering it from the stats of the database – something TokuMX can’t do, since it doesn’t have accurate stats. So what TokuMX does instead is iterate over all entries, which with 8m entries will take a while. I worked arround it to get kingboard at least up and running, by calculating the total from some aggregated values that i was loading for the page anyways. (I’m not sure if that trick will work in all places though).

Once kingboard was running again, i decided to check the speed of which i can load mails into it from our message queue. Running 20 instances of the importer, it was about 26% faster (more mails per minute) than the same importers when they where writing to MongoDB.

Surfing arround a bit while the import was running, i noticed that:

  • most pages would load with no noteable speed difference to MongoDB.
  • those pages that did inline map/reduces felt alot slower. It turned out that during the import for whatever reason a few of my indexes where not imported. doing those in a running instance by .ensureIndex() took ages (complete night). Once those indexes where set it was faster than before, but still felt a bit slower than with MongoDB, i haven’t found why yet.
  • EDIT: not only the inline map/reduces are slower, it turns out that proper map/reduces are a lot slower as well – for example right now i have an map/reduce op running since 8 hours, which is in the emit phase, at about 38% now (thats with 900k documents to be mapped with an average of between 4 and 5 emits per document)
  • Skipping kills. if i want to go to page 300,000 on the kills i have to wait for some minutes. I haven’t looked at the source for this one yet, but i imagine its because of doing something similar as in the count(). MongoDB actually is quite slow there as well, but we are speaking 15 seconds to 4 minutes. Again, i haven’t looked at the source yet, and right now i don’t see a quick win here on how to fix this. (Turning of Pagination sounds like the wrong approach, but maybe paging by hours or days instead of 20 kills a page?) EDIT: i’ve talked to someone on IRC about it and apparently my hunch was right, skip i doing similar stuff like count there because TokuMX can’t do some optimizations MongoDB is doing here.
  • EDIT: even with an index on totalISKValue this takes ages:
    $data = Kill::find()->sort(array(“totalISKValue” => -1))->limit(12);
    EDIT (AGAIN): This was caused by twig doing a count() before looping on the list, which was basically just the cursor. Count’s default behaviour in MongoDB (and tokumx) is to count all elements and ignore the limit. (optional boolean parameter allows to reverse behaviour). So basically this problem was just a copy of the original count problem.
  • At one point during a huge update_stats (which did update stats for approx 1m kills, something that in normal circumstances shouldnt happen) the oom-killer of my host system actually killed TokuMX’s mongod instance. (have to look into settings for that, this is not a TokuMX issue)
  • i had a few times a MongoConnectionException “Failed to connect to: localhost:27017: Remote server has closed the connection”, and i’ve yet to figure out why.
  • The Data on Harddisk shrunk from about 63GB to 11GB.

As it seems its not having all indexes it should have yet, and there are a few other tweaks that i’m going to make in the next few days to work more nicely with TokuMX, however i’m really hoping more people adopt TokuMX so maybe it gets a bit more love to grow to the solution it could be.

Advertisements

2 thoughts on “Kingboard & TokuMX for MongoDB

  1. I’ve been playing with Solr, with 8 million kills I got from Karbo. If you’ve not run into it, it’s a web service based off the lucene search engine. Runs well, searches quickly and as I’m not storing the untokenized data, the index is less than 1GB in size. possibly something to consider.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s