Find solution
Are There Any Databases Of Digitalised Natural Stimuli On Milliseconds Scale?

Which analytics databases scale *down* as well as they scale up on the cloud? In other words, is there a BI platform that supports cloud elasticity natively?

  • Cloud providers charge per hour of use, even if the instances are idle, so ideally one would shut down most nodes when the cluster is not in use. Most analytics and/or columnar databases require complex data redistribution sequences in order to scale up or down. Are there any that don't, for example where the data would permanently stay on S3 and EC2 instances could be spawned accordingly based on load?

  • Answer:

    I disagree with the Hadoop solution suggestion. Hadoop is designed for data analytics across an upwardly scaling hardware infrastructure. Its best use case is for unstructured large data sets. Unless you're going after unstructrured data sets, a more traditional relational database set up is going to be more cost effective.   It sounds, based on your comments regarding cloud price, that your problem isn't technology - You don't build churches for Easter Sunday and you shouldn't build databases for peak performance, you build it for average performance and set expectations with users that more volume means slower performance OR you build it for peak, but use commodity or IaaS hardware.   Does it have to be a cloud solution? If it were me building the solution, I'd look to bring it in-house (if it made sense - you have to have a strong infrastructure and data center for some of the technology out there). That way you have more control over the architecture, and if the peaks are known well in advance, you can even plan and scale for them.   If it has to be a hosted solution, I'd look at Adobe, Amazon, or even IBM and see if they can meet your needs. You might even look at Teradata, they do their own hosting as well. You probably have a better shot at getting a pricing solution to meet your needs.

Patrick Pitre at Quora Visit the source

Was this solution helpful to you?

Other answers

Looks like Hadoop is what you are looking for but minus the complexity.  I recently sat through a deep dive on the Cloudera offering around Hadoop.  Looks like they can help reduce the the monitoring, management and other implementation related overhead.  http://www.cloudera.com/

Mayank Misra

I'm only going to partially answer this question, mostly for the sake of brevity.  The elasticity requirement is usually driven (primarily) by a performance + cost equation.  If you're open to other scenarios - ex. not strictly "tier 1 cloud" - then http://Infobright.com offers a compelling solution and price point as either an on-premise; hosted or managed solution. Full Disclosure: DecisiveBI offers the latter two Infobright capabilities to clients.

Matthew Caston

In terms of what Amazon Web Services offers I think you may find Amazon RedShift meets the "scale-down" / "scale-up" criteria, as well as manages redistribution and "snapshots" of the cluster that can be replicated or shared. While RedShift doesn't dynamically scale up based on load, you can resize it while it is online and it remains it "Read Only" mode while it copies data to a new cluster, then cuts over to the new cluster. Further, you can load a large cluster, create a snapshot, delete it, and then bring it up long enough to run whatever you need, and then scale it down again. All this could be automated and orchestrated to using a combination of services like AWS DataPipeLine and RedShift. That's how I'd approach this scenario.

Brian McCallion

This is possible with Vertica on our AWS based platform, elasticBI. The trick is to scale up the Vertica cluster in a controlled manner, and even more controlled when down scaling (you are essentially allowing k-safety in Vertica to redistribute the data). This does take time to redistribute, and is not something you can do frequently (like intra-day). It may be possible also to combine hadoop for basic map reduce, and then use a Vertica cluster to manage the scaling on a smaller aggregated dataset. That said, being on AWS does give you some elasticity, if your peaks and valleys are known and arent granular to the point of doing the scaling intraday.

Rohit Amarnath

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.