How to remote debug my application with an IDE?

How do I debug my map-reduce program while using IDE?

• - I wrote my Map and Reduce in Intellij IDEA - my hadoop is running in pesudo-distributed mode - I want to run the program for a small set of data through IDE to make sure everything works - Is it that I have to submit the job through command-line and watch hadoop logs? - any better ideas on it?

• Answer:

  I strongly recommend using something like MRUnit[1] (which supplies the necessary test infrastructure to mock out Hadoop) to test map and reduce classes rather than attempting to test jobs in the context of the framework itself. Remember you want to unit test functionality and correctness and using integration testing in a small (but real) cluster to prove the glue between your code and that of the infrastructure. Both mappers and reducers should be self contained (i.e. have no externally visible side effects) which makes testing trivial (in theory). In other words, you should be able to pass a mapper a key / value pair and get the intermediate key / value pair you expect. The same is true for your reducer. You don't need the rest of Hadoop to prove to you that works. Draw a clear line between what your code does and behave according to the contract of the system and integration testing should really just be about throwing real world data at the code. When you find a record that breaks your code, copy it into your MRUnit test input dataset, masking sensitive fields accordingly, and now you've started building future regression validation against known problems. If you're test input dataset in unit tests is over ~100 records, you're probably over thinking the problem. Once unit testing is complete, build a set of larger, real world datasets that run on a test cluster. Use something like to drive tests and validate results regularly (e.g. hourly or when a new build completes). [1] http://mrunit.apache.org/