How do I use clojure-hadoop correctly with an existing hadoop cluster?
-
I want to be able to submit a jar. There is a "hadoop" bin on the cluster that prints the following: Usage: hadoop [--config confdir] COMMAND where COMMAND is one of: namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode datanode run a DFS datanode dfsadmin run a DFS admin client mradmin run a Map-Reduce admin client fsck run a DFS filesystem checking utility fs run a generic filesystem user client balancer run a cluster balancing utility jobtracker run the MapReduce job Tracker node pipes run a Pipes job tasktracker run a MapReduce task Tracker node job manipulate MapReduce jobs queue get information regarding JobQueues version print the version jar <jar> run a jar file distcp <srcurl> <desturl> copy file or directories recursively archive -archiveName NAME <src>* <dest> create a hadoop archive daemonlog get/set the log level for each daemon or CLASSNAME run the class named CLASSNAME Most commands print help when invoked w/o parameters. I will need my first mapreduce to read from a file on an nfs share that the whole cluster can access, and write to hdfs, and later mapreduces will map hdfs data to hdfs data only. I will also need to specify the mapper, reducer, and (wrapped) reader functions on the command line, because I couldn't figure out defjob properly. If you can tell me how to use defjob correctly instead, that would be fine as well.
-
Answer:
Oh I solved this a long time ago. Turns out you put a call to gen-class in your mapreduce code, so it'll get compiled to classes and put in the jar, then bundle it with clojure-hadoop-job.jar or similar (lein uberjar is good for this), and call it with hadoop jar clojure_hadoop.job <options> where <options> are options for clojure_hadoop.job (so, -input, -output, -map, -reduce, etc.).
Leif Walsh at Quora Visit the source
Related Q & A:
- How should I use Youtube API?Best solution by Stack Overflow
- How do I use "John et. al. 1990 " citation for more than two authors?Best solution by tex.stackexchange.com
- Where do I find my Yahoo briefcase and how do I use it?Best solution by Yahoo! Answers
- How do I add a new stock to an existing portfolio?Best solution by chegg.com
- How do I use chat when I am in a room?Best solution by Yahoo! Answers
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.