Find solution
Should "Big Data" be capitalised?

Are "big data" and data analysis practically distinct?

  • "Big data" seems to focus on accessing and serving up big quantities of data quickly and efficiently. Case in point: talking to an engineer at a purported "big data" firm I was surprised when the concepts of QQ plot and alpha-stable distributions were met with a blank stare. Is this an isolated data point or can one attest knowledge in "big data" limited to data structures and algorithm design but largely exclusive of probability and statistics? Is big data doing serious analysis or is it a buzzword for scaled IT with a splash of visualisation?

  • Answer:

    Yes I think your right.  My experiences are similar to yours.  I think the reason is they are actually quite distinct, one group(usually tech) manages the data storage and the other group(analyst) study the data.  I thought this was odd at first also but if you think about it most companies have similar structures with other proceses(for example at a asset management firm, you wouldn't ask an IT person how the firm calculates their NAV just because the data is stored on a server they manage). Big data is largely a technology thing, I know a few hedge funds using this technology and although I'm sure the IT people are putting it on their resume, I can assure you that the analyst are indifferent to the type of data as long as it gives them the information they need. The way I see it, Data Analysis is about getting meaningful information from data while the people running the technology's job is to make sure the data is scaling, available, and as fast as possible.  To make more group responsible for the other would just cause a loss in focus.

Gautam Bakshi at Quora Visit the source

Was this solution helpful to you?

Other answers

My guess is that you received a blank look because you were talking to an engineer.  Big data consists of volume, velocity and variety of data.  Engineers deal with volume and velocity; how do we stole all of this data.   Data scientists/analysts deal with the new variety.  These new data sets consist of structured, semi-structured and unstructured data.  In order to do any real analysis the data needs to be processed and given a structure.  Current big data analytic trends include dealing with data sets that are too large to be handled in a traditional EDW.  Most current EDW analytics are done in memory.  This restricts the amount of data which can be used.  For the most part enterprise has skirted this issue by sampling the data and developing models from the sample set.  For the most part this works just fine.  If, however you are looking for subtile fluctuation in the data this becomes much more difficult.  As you probably know, large amounts of data tend to smooth out the data set.  This means that small fluctuations that are observed when using the entire set are much more likely to be real rather than noise.   This gives companies the ability to look for small changes that can optimize returns.  Moving away from the traditional analytics, big data architecture allows companies to analyze computer logs and other machine generated data.  (look up Splunk they are amazing) This type of analysis can predict when machine are likely to fail.  It can also give insight into optimizing load balancing and other types of tuning that was previously only accomplished by an engineer turning a knob until the box purred.  The holy grail of big data is to plug into all the crap written and published on the internet via social media and blogs then turn this data into intelligent, fiscally provident decisions.  For the most part this is an uncracked nut but the tools are there to make it work.

Robert Eckhardt

I agree with the post above, "big data" is a very broad term, and reflects a mix of skills in data engineering vs. analysis. There are also many different flavors of analysis to consider, e.g. statistics vs. probabilistic models. I've tried to explain the different skills here:

Anmol Madan

The two are completely different.  Here are two blog posts that, I hope can help you! See the "Elephant and Rider" Analogy @ http://bit.ly/131LojP See more on the state of affairs @ http://bit.ly/Wlesi4 Analytically Yours, Bruno

Bruno Aziza

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.