The way that pilot/alliance/corporation/general stats where generated wasn’t exactly what i would call optimum. MongoDB does not allow to split up map/reduces over several processes (yet) (except with several server instances), so a map/reduce can take quite long.
So far Kingboard was caching map/reduce results to avoid it to have the high runtimes with large amounts of killmails (i tested with up to 6mio killmails).
This would have caused everyone who actually had a cache miss (expired / not in cache yet) to have unacceptable high wait times.
Now the refresh of the cache could be asynchronous to the request (deliver expired cache while triggering refresh) – but the initial miss would still be there.
Another solution to the problem would have been to autogenerate the stats regulary, however to do that for all sorts of stats by allways map/reducing the full data.. well that gets quite slow too.. so stats would have been generated every two hours or so (well maybe one)).
Thank god MongoDB 1.8+ allows for ‘reduce output’, which basically means the result of a reduce saved in a collection will be auto reduced with the result of an reduce. That makes it quite easy to do incremental map/reduces, you only map the new data, reduce it, mongo takes the old stats reduces the old stats + the new stats, and tada you have the result. With this the run times of the map/reduce get quite manageable, so in my test install im generating all stats every 3 minutes atm (could go as small as every 5 seconds if i wanted to).