What's the best machine for development?

How is industrial, product-oriented machine learning software development different from academic machine learning research or industrial research prototype development?

• I'm interested in differences like "we use simpler algorithms," "we have to highly optimize for running time," "we more carefully architect the code to adhere to overall design guidelines," "We spend more time debugging," "We develop new algorithms much less often," "We take direction from management rather than exploring our own ideas," "we spend most of our time doing things other than machine learning development" etc. Basically, I want to know how academia or prototype-driven industrial research is different from a machine learning shop in a place like MS Bing, Google, Apple, Netflix, etc.

• Answer:

  already provided a fantastic answer. Let me add some more major differences. In industrial ML development, usability (UX) is a very important factor. Not so much in academic research. This is the same as the general industrial software vs academic software comparison. Industrial ML development tends to focus on implementing the existing state-of-the-art algorithms and dealing with software design issues. Developers typically don't have time budget for and discouraged from investigating a new ML algorithm. Academic researchers, on the other hand, need to work on new ideas so that they can publish. Academic research is driven by competitions or challenges with fixed and typically clean datasets and with fixed tasks. Ultimately, research needs some metrics to be deemed successful. In reality, data almost always evolve and typically need cleaning. Even the data schema can change frequently. So, a lot of industrial ML development effort may not be on the core ML algorithms but on data sourcing, formatting, cleaning, features engineering, etc. Even the objectives may evolve as well. For example, with the same collected data, your objective can change from a simple exclusive classification to multilabel or hierarchical classification. ML research is (almost) all about accuracy. Most academic papers mention only accuracy and no computational cost. This is partly driven by the competitions culture mentioned earlier. Even if a paper mentions the impressive train/evaluate time, what matters more is whether it is the top performer on a popular scenario (e.g. MNIST or ImageNet). Note that this is not typical in other research areas. In Numerical Analysis (my former background), for example, accuracy and computational cost are always reported together. In industrial ML work, computational performance is absolutely critical. About (ML) extendability. Industrial ML development can be classified into two categories: generic ML software for people to use (e.g. RapidMiner, Mahout, etc.) and software to perform really well on specific tasks of high impact (e.g. Web Search, Image Search). The former needs to be extendable (i.e. being able to solve multiple tasks, with multiple data formats, from multiple data sources, etc.), much more extendable than academic software. There are actually a lot more differences, which I'd be glad to discuss if interested.