Why Hive's Sort Merge Bucket Map Join does the join in just one mapper?
-
I have a Hadoop cluster and I use Hive for querying. I have two large tables, which are bucketed and sorted on the join key. So, I think using "sort merge bucket map join" would be helpful. I set the required flags and then run the query. I see that it starts a job which has 1 mapper and no reducers. Question: Why Hive does the join in one mapper? Did I miss anything? Because tables are bucketed on the join key, it seems that join can be done in multiple mappers in parallel. Why Hive doesn't do that?
-
Answer:
It's tough to say without taking a look at the explain plan and/or the mapreduce job's job.xml because the answer here may depend on a few things depending on your hive configuration, the format of your data set, the metadata associated with it among other things. 2 things that jump out of the top of my head are: 1. InputFormat: Set by hive.input.format property, it may be a format that combines the buckets when sending them to the mapper. 2. Size of your dataset and buckets: If the bucket sizes are small, they may be bundled together before being sent to mappers.
Mark Grover at Quora Visit the source
Related Q & A:
- How can I merge all my Google calendars into my primary one?Best solution by Quora
- How to merge one yahoo group with another yahoo group?Best solution by Yahoo! Answers
- Does anyone know where I can buy just one issue of Italian Vanity Fair here in the US?Best solution by Yahoo! Answers
- Can someone be colored blind in just one eye only?Best solution by Yahoo! Answers
- Why the best French basketball players never join the French National Basketball Team?Best solution by Yahoo! Answers
Just Added Q & A:
- How many active mobile subscribers are there in China?Best solution by Quora
- How to find the right vacation?Best solution by bookit.com
- How To Make Your Own Primer?Best solution by thekrazycouponlady.com
- How do you get the domain & range?Best solution by ChaCha
- How do you open pop up blockers?Best solution by Yahoo! Answers
For every problem there is a solution! Proved by Solucija.
-
Got an issue and looking for advice?
-
Ask Solucija to search every corner of the Web for help.
-
Get workable solutions and helpful tips in a moment.
Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.