Find solution
How can I query Parse data by creation date with Swift?

When does big data become small data again?

  • Big data. All the buzz for some time now. Terabytes of useful data, so rich that in many cases, ad-hoc analysis requires asynchronous computing. Launch your query, get results several hours later. Or have a regular pre-processing algorithm run on the data regularly to cache the results you'll need to access on an ad hoc basis later. But will this always be the case? Maybe today's terabytes, not manageable in real-time, could possibly be digested in a fraction of a second someday? Returning instant results like we can do today with, say, a few megabytes of data. For instance, I imagine Amazon has a quite impressive data warehouse on its customers' purchasing behavior. They probably have processes running for hours through the data, extracting patterns and clusters, which are then translated into smaller databases, used transactionally (in real time) whenever I go to the website and they make a recommendation for me. One day, maybe the warehouse will be the regular database, and every time I go to Amazon, the entire clustering / pattern detection will be ran real-time on the whole data? Correct me if I'm wrong on my understand of today's situation. If not, my questions to the tech experts out there are: - are there precedents in the computing history of such a leap? similar patterns of "impossible, would take a year at full power" one day, to "get the output in real time" the next day? I imagine there must be - what kind of time horizon are we talking about on these historical examples? - what kind of time horizon can be envisaged (based on history, or other considerations) for today's terabytes to become manageable in real time, with no pre-processing, as I described above? - will we hit some kind of limit in data processing (whether linked with sheer processing power, or data retrieval mechanisms, networks, etc.) or can we trust Moore's law indefinitely? Please note that while I have a tech background and a more or less scientific mind, I'm not at all up-to-date on pure technology considerations (as you can probably deduce from my non-technical wording above :) ). So please answer in simple terms ! Thanks in advance! EDIT: see comments to Tom's answer for a more precise wording.

  • Answer:

    Big data becomes small data when we consider what's coming next...big video data. Intel services are already harvesting data from video at 1000 frames per second x 2500 discreet pieces of data per frame. That dwarfs what we consider big data and sets up the next round of discover/innovation...the ability to take video analysis and figure out body language and a host of other very interesting things.

Chris Taylor at Quora Visit the source

Was this solution helpful to you?

Other answers

So I'd cooperatively challenge the paradigm that says you ask and then wait several hours - more often than not that is indicative of a poorly designed data structure, job and/or cluster. What you are asking about can be accomplished in at least some patterns today. For example - there are Streaming solutions (including ours) that change the paradigm on how this can be approached. In fact I was with a customer yesterday and we sketched out a fundamental shift in approach which is to us Streaming solutions to do the ETL in memory and only store once the data was ready to be consumed in HBase. That removed both latency on ETL and on access. Hardware costs are going to be reduced by 80%+ with this approach while improving performance. Another option is storing in HDFS and using SQL constructs to access the data without MapReduce, but more on that later. Hope that example helps. If you lay out your specific problem it is likely there are a number of other approaches that can be taken

Tom Deutsch

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.