Find solution
What is data processing?

What is the best approach to perform in line processing for Big Data?

  • I have below requirement: 1. Fetch data from different sources (like twitter, blogs) and load into MongoDB. I am planning to use Flume for this but open for other options 2. Need to perform some calculations on incoming data like Sentiment of text (I am using simple lexical approach).   Couple of options I thouht are (open for any other option): 1. Perform these calculations in Flume using Interceptors and load results into MongoDB 2. Load data into MongoDB and then use Strom with Mongo-connector to perform calculations.   Please suggest me the correct approach.

  • Answer:

    It does not matter much in the beginning and will have to changed later on anyway. Side idea: if you are looking for an in-line solution, you can go with a REST-ful service that does the processing and stores already processed data. This may happen in parallel with the raw data you are (probably) storing now. You can even write to a separate storage to ensure they don't interfere. The advantage of this approach is that, since processing is stateless, you could always spawn more machines that do the processing itself and save on setting up the workflow, at least at the early stages.

Dima Korolev at Quora Visit the source

Was this solution helpful to you?

Other answers

I built a system doing exactly what you want to do @ jkool. What we did is stream data into Kafka or JMS and pipe it into a STORM cluster running CEP engine where all calculations are performed on streaming data. Raw data as well as computed data can then be stored in MongoDB or other DB.It all depends how quickly you need to process incoming streams. The approach above can clock at many hundreds of thousands per second + depending on cluster size.

Albert Mavashev

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.