How will big data transform the practice of software development?

Big data changed software development years ago when it comes to apps, i.e., data-driven APIs (or "headless apps") that are called by many apps today. Think about what's behind the most popular APIs. The companies who donate the most code and participate most in Big Data open source communities--many of the largest Web properties such as Yahoo, Google, LinkedIn, Twitter, Facebook, and Netflix, among others--all run big data-driven environments, and all offer popular APIs. Google Analytics was launched in 2005, and the Google Analytics API has been around since 2008. Facebook Graph API was introduced in 2010. Data-driven APIs are major revenue generators for large web properties. Forbes in a January 2014 article pointed out that 90 percent of Expedia's revenues were generated from APIs. (See http://www.forbes.com/sites/mckinsey/2014/01/07/ready-for-apis-three-steps-to-unlock-the-data-economys-most-promising-channel/.) APIs are of course one enabler for more agile developments.but there are others. These web companies are the very same companies who are central to the DevOps/continuous delivery movement. They all run heavily instrumented environments and routinely mine their logfiles for clues on how to optimize their services and release code more frequently. They focus a lot of attention on rapid feedback-response loops throughout their organizations and constantly tune their sites and other services. Vendors such as Splunk and Sumo Logic that specialize in big data logfile analytics have many of these companies as customers. We did a complete analysis of DevOps, antifragility and continuous delivery oppportunities in 2013 that's available for download here: http://www.pwc.com/us/en/technology-forecast/2013/issue2/index.jhtml Perhaps what you're identifying is an expertise gap between large web companies and other enterprises. Many large companies in other verticals and independent developers aren't plugged in to the OSS environments or find tools like MapReduce strange or think they're too risky. But those who run Hadoop + NoSQL clusters internally and staff or nurture the right roles can certainly develop beneficial big data apps, and tools such as YARN are making it possible to develop apps using a variety of languages, and many major analytics platforms are incorporating support for Hadoop. See some of the decks from Hortonworks or Cloudera on YARN for more info, such as http://www.slideshare.net/cloudera/introduction-to-yarn-apps. These companies can certainly instrument their own online environments and mine the data, or build sandboxes to mine the public web for info about markets and how their brands are perceived in the marketplace, and many companies have done this. A number of high-tech manufacturers, transportation companies, utilities and energy companies are well along toward big data analytics, if not there yet. They may not have taken the step of using big data to drive software innovation yet. A lot of the activity is behind the scenes, and the public at large is not aware of it, but info on what's happening in this area is certainly findable on the public web. If you need a backgrounder on big data techniques and the business implications of the technologies, back in 2010 we put together a report for the benefit of large enterprises who wanted to take advantage of Hadoop and NoSQL: http://www.pwc.com/us/en/technology-forecast/2010/issue3/index.jhtml

The biggest application I've seen have been in the use of big clusters to handle large bodies of generated information. My guess would be that the downstream effect of this is that software designers will have less fear of generating large bodies of data as the output of their processes. People have a nasty habit of generating data, not because it's useful, but because it's possible. They they assume that someone will be able to handle that data. The move to big data processing like Hadoop will make large data processing tasks easier and, in turn, will generate more large data tasks.

Recently i came across an article that helped me tremendously in understanding a Big data's importance for software developers . I suggest you also to read it once .http://www.greycampus.com/blog/big-data/importance-of-big-data-for-software-developers?utm_source=quora&utm_medium=Referral&utm_campaign=sma-ibdsd