How to correctly use ScheduledExecutorService?

How do I use clojure-hadoop correctly with an existing hadoop cluster?

  • I want to be able to submit a jar.  There is a "hadoop" bin on the cluster that prints the following: Usage: hadoop [--config confdir] COMMAND where COMMAND is one of: namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode datanode run a DFS datanode dfsadmin run a DFS admin client mradmin run a Map-Reduce admin client fsck run a DFS filesystem checking utility fs run a generic filesystem user client balancer run a cluster balancing utility jobtracker run the MapReduce job Tracker node pipes run a Pipes job tasktracker run a MapReduce task Tracker node job manipulate MapReduce jobs queue get information regarding JobQueues version print the version jar <jar> run a jar file distcp <srcurl> <desturl> copy file or directories recursively archive -archiveName NAME <src>* <dest> create a hadoop archive daemonlog get/set the log level for each daemon or CLASSNAME run the class named CLASSNAME Most commands print help when invoked w/o parameters. I will need my first mapreduce to read from a file on an nfs share that the whole cluster can access, and write to hdfs, and later mapreduces will map hdfs data to hdfs data only.  I will also need to specify the mapper, reducer, and (wrapped) reader functions on the command line, because I couldn't figure out defjob properly.  If you can tell me how to use defjob correctly instead, that would be fine as well.

  • Answer:

    Oh I solved this a long time ago.  Turns out you put a call to gen-class in your mapreduce code, so it'll get compiled to classes and put in the jar, then bundle it with clojure-hadoop-job.jar or similar (lein uberjar is good for this), and call it with hadoop jar clojure_hadoop.job <options> where <options> are options for clojure_hadoop.job (so, -input, -output, -map, -reduce, etc.).

Leif Walsh at Quora Visit the source

Was this solution helpful to you?

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.