How can I clear the mongoDB oplog?

Is anyone logging the applications logs to MongoDB? Is it possible to run real time analysis/monitoring jobs on the log objects in MongoDB?

• I am currently looking into an option of writing application logs to MongoDB using  log4mongo-java. Is any one logging the applications logs to mongoDB? Is it possible to run real time analysis/monitoring jobs on the log objects in MongoDB?

• Answer:

  I've seen at lot of companies are using MongoDB to store application logs. Its schema-freeness is really flexible for application logs, at which schema tends to change time-to-time. Also, its Capped Collection[1] feature is really useful because it automatically purges old data to keep the data fit into the memory. People aggregates the logs by normal Grouping or MapReduce, but it's not that fast. Especially MongoDB's MapReduce only works within a single thread and its JavaScript execution overhead is huge. New aggregation framework[2] could solve this problem. Another concern is high write through put. Although MongoDB's insert is fire-and-forget style by default, calling a lot of insert command causes a heavy write lock contention. This could affect the application performance, and prevent the readers to aggregate / filter the stored logs. One solution might be using the log collector framework such as [3], Logstash[4], or for batch + asynchronous insertion. Disclaimer: I'm a committer of Fluentd project. [1] http://www.mongodb.org/display/DOCS/Capped+Collections [2] http://docs.mongodb.org/manual/applications/aggregation/ [3] http://docs.fluentd.org/articles/apache-to-mongodb [4] http://logstash.net/docs/1.1.0/outputs/mongodb