Will there be memory problems between multiple web service requests?

What are the ways to re-architect a Hadoop MapReduce job which requires loading of large models into RAM to facilitate processing?

• Im working on a MR job which processes real-time data from various sources.  For the processing, each map task requires loading of some GBs of resources into memory. This memory overhead is limiting the number of map and reduce tasks that can be on a machine in parallel as each map task is run in a different JVM instance by Hadoop and each map instance loads its copy of resources. Is there a way where all map tasks can access the resources from a shared memory? If not, would a web service based model (hosting the memory dependant module as a service on a different machine/s) scale for  processing 10s of millions of requests per day?

• Answer:

  You would be better off not using MapReduce.  The problem statement has all the classic signs of trying to shoe horn an algorithm into MR when other frameworks would be a much better fit.  If you wanted to stay within the Hadoop universe, then take a look at YARN as part of the Hadoop 2.x code base.  You'll need to write your own Application Master, etc, to do what you are wanting to do.  But it seems to me that is a better answer.