How to add a new table to a data source?

Business Intelligence: How is it possible to deliver real-time, self-service BI with a legacy RDBMS source?

• Traditional BI solutions require processing time to load source data into a fixed-schema data warehouse.  But BI consumers (decision makers & analysts) want to build their own dashboards and reports without being limited to a pre-defined schema. A key principle of traditional ETL solutions is to extract only the data that is needed (to populate the pre-defined data warehouse schema) from the source.  If our extracts are lightweight, we can regularly pull from an OLTP source in near real-time.  But copying every column from every table in that source, in order to make ALL of that data available to the BI consumers, is going to take a LOT more time.  And it's going to create a load that the OLTP server often can't handle well. I'm interested in the work that's being done to enable real-time analytics against big data stored in Hadoop.  But it seems that everyone advocating for real-time, self-service BI "without limits" is assuming that the source data can be pumped directly into Hadoop.  But when we don't control the data source, what good is real-time analytics on top of Hadoop if it's impossible to get the data INTO Hadoop in real-time?

• Answer:

  I find it very hard to believe that anyone who is advocating "real-time self-service BI without limits" and "Hadoop" has any idea what they are talking about. I would also question the business value of the real time analytics desired.  What key business decisions need to be made on a real time basis?  Why is yesterday's data not good enough?  If your decision maker actually got the data in real time, would they be able to act on it in real time?  99.9999% of the time, the answer is no. I also am not sure how exactly one would intend to build reports and dashboards without a schema.  At some point, someone somewhere has to define metrics and attributes, regardless of what tool you use.  Whether that is done as a function of ETL or at a logical reporting level, a structure must be placed on the data in order to make it consumable. In addition to that, Hadoop is not a solution built for real time.  Hadoop/HDFS can store and process a ton of data, but most certainly not in real time.  The Map-Reduce framework is a batch processing tool, not tool that can instantly grab and crunch data. To address how this can be done from an RDBMS (there is nothing legacy about it), there are products out there today such as Golden Gate which are intended for this purpose. For the very few use cases that truly do require real time analytics, most companies will use a data replication tool in conjunction with an in-memory solution.  This, however, typically costs a ton of money (to build and operate) and simply doesn't make sense for most use cases.