Find solution
Should "Big Data" be capitalised?

Does big data threaten the traditional data warehouse / business intelligence model / stack?

  • The traditional data warehouse / BI paradigm forces companies to invest upfront time and effort defining data models that accurately reflect the way their businesses work (build a data warehouse), so that when a business user has a question the data is already structured in way that lets her easily interrogate it and generate answers.  In contrast, users of big data typically only extract and structure the data after they formulated the business question, enabling them to extract only the data relevant to answering their question from the big data pool.  A second business user, asking a different question, might create a totally separate data mart to answer that 2nd question, and extract a totally different set of data from the big data pool to do so.  Is this a threat to the future of the data warehouse as the single source view for all business users?  (And the future of business intelligence stacks as the tools for business users to access that view?)

  • Answer:

    I think you are missing a key part of the BI paradigm - self service.  A properly designed Business Intelligence system is designed to be used by consumers of the data for both ad hoc queries and canned reporting.  One of the beautiful things about Kimball Methodology is that the Star Schema is immediately intuitive to people who may understand how to use data, but don't necessary understand data structures. Even though there are packages like Hive for Hadoop, the NOSQL databases are still a long way away from having the breadth of end-user oriented tools that traditional Relational Databases have for them. The key to building any system is to understand that there is a time and place to use any toolset.  Being pragmatic is always better than being dogmatic.

Steve Larrison at Quora Visit the source

Was this solution helpful to you?

Other answers

I think they do two different things.  An EDW solution allows users to write algorithms that give very precise answers.  The downfall is that there algorithms tend to be computationally intensive which limits the amount of data that can be churned through.  Big data has the ability to churn through huge volumes of data but precision is sacrificed.  Big data is highly effective for trend analysis, improving customer experience and other types of process.  EDW type analytics are good for fraud detection, predicting values with granularity down to the unit, etc.  These ought not be thought of as an either or scenario.  Both work well in conjunction.  Big data platforms can reduce the workload of EDW and increase efficiency in trending type tasks.  Big data platforms can also narrow down where to look for things allowing the analyst to refine the EDW algorithms further because the data load will be reduced. Regarding the second part of your question.  EDW and Big data platforms aren't getting their data from the same pool.  EDW data is typically structured transactional or master data.  Big data platforms are primarily pulling unstructured and semi-structured data into the mix.

Robert Eckhardt

Also inherent in your statement is I think a misconception that both big data and EDW's rely on a single version or view of the data. That concept is an Inmon concept that is proliferated because of the lack of good data and BI governance. As long as there is strong governance in place, there is nothing wrong with having multiple versions of data, in fact, in larger shops it might even be necessary as a single version becomes expensive to maintain with large data sets and competing business priorities.   Big Data on the other hand shouldn't even have a single version of data concept, as the data is unstructured and the use cases are more often defining what the single version should be as opposed to publishing what it is.

Patrick Pitre

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.