MapReduce Debugging with Jumbune

Report
MapReduce Debugging
with
Jumbune
*
Agenda
Zero Tolerance in
Production
Debugging
Challenges
Jumbune’s
Debugger
Debugging
MapReduce
*
Zero Tolerance in Production
Typically, working in Big Data we Ingest and analyze multi-terabyte to petabytes of
data on multi-node cluster to perform actionable analytical outcomes for
• Discovering opportunities and solutions
• Deriving operational intelligence
• Driving sales
Production errors and failures results in significant loss in revenue and time.
Enterprise analytical solutions executing in production have zero acceptance to
bugs & errors.
*
Debugging challenges
Huge applications, the symptom and cause
may be in remote parts of the program.
Multiple Components that work in tandem
may trigger rare or difficult to reproduce input
sequence, program timing.
Complex systems - Flaws due to human
mistake or misunderstanding and is difficult
to trace
Must be frugal, scalable yet detailed
*
MapReduce Debugging
●
●
●
●
●
●
●
●
Handle Billions of <Key, Value> pairs
Through multiple phases and components
On thousands of machines
Customized logic in Mapper, Reducers, UDFs
Frugal that it does not escalate the execution time
Detailed enough to let the developer understand
Scale to terabytes of data
Scale to thousand node clusters
*
Jumbune’s Debugger
*
Dynamic
instrumentation
Submits the job to
Jumbune for flow analysis
jumbune
MapReduce
execution flow debug
results
Job executed on the
cluster
The Developer develops
chained & complex
MapReduce application
Log parsing and analysis
Logs collected from the
executed cluster nodes
*
Asymmetric Advantages
Presents easy to understand hierarchical execution flow details of MapReduce Job
Bring down hours of execution logic debugging trails by identification of root cause within minutes
Verify execution on all participating nodes of the cluster
Ability to work with all major Hadoop Distributions
xxx
xxx
MapReduce Dev
Logic Test
*
Hierarchical Flow Analysis
Trace <Key,Value> pairs into each control structure in
every phase of MapReduce
Regular expressions and Custom Java validations on
every phase
Map
Method()
IF1
Job, phase and instance level details
Method, counter and control structure details for deeper
analysis
IF1
IF2
IF1
Input keys, output records and filtered in/out details for
advanced debugging.
IF3
Method()
Chained job support
Method()
IF2
*
Let’s debug your Jobs together!
Website
• http://jumbune.org
Contribute
• http://github.com/impetus-opensource/jumbune
• http://jumbune.org/jira/JUM
Social
• Follow @jumbune Use #jumbune
• Jumbune Group: http://linkd.in/1mUmcYm
Forums
• Users: [email protected]
• Dev: [email protected]
• Issues: [email protected]
Downloads
• http://jumbune.org
• https://bintray.com/jumbune/downloads/jumbune
*

similar documents