Quantcast
Channel: Leadership Experience from Trenches - Big Data
Viewing all articles
Browse latest Browse all 9

Our first Map Reduce Program in C#

$
0
0

As my first .net/C# program, I am going to follow the tutorial written as part of HDInsight server installation. The diagram below nicely depicts our approach. I really liked the diagram “Visual Objective”. It nicely depicts the model.

Step 1:

Save log file locally. Push/load log file (to be analyzed) to HDFS. Here are the commands.

clip_image001

Step 2:

Create a new VS 2012 project (Class Library) and add “Microsoft.net Map Reduce API for Hadoop” NuGet package. Don’t forget to add “using Microsoft.Hadoop.MapReduce;” in your code file.

Step 3:

Create Map and Reduce class that extends MapperBase and ReducerCombinerBase classes respectively. Here is how skeleton look like.

clip_image002

We would also need a job definition that could be submitted as Hadoop Job.

Step 4:

Build project and output Lesson1.dll (name of my project). Copy this dll to subfolder MRLib under Debug\Bin directory (not sure whether I should do in future too or give a path of the dll). In the command prompt, go to MRLib directory and run the command as shown in the image below.

clip_image003

Hurray!!! Made me happy to see the output :-) My first hadoop Map Reduce program ran successfully. In next days, I will also be converting some existing java examples into .net before I take a real life problem. Stay tuned!!


Viewing all articles
Browse latest Browse all 9

Trending Articles