Hadoop Training

You can explain the architecture of distributed storage and spin up a pseudo-distributed Hadoop instance on your local environment.

1. What Big Data is and the challenges it brings90 min
   • Volume, velocity, and variety constraints
   • Vertical vs horizontal scaling limitations
   • Cost walls of SAN and NAS infrastructure
2. The history and idea behind Hadoop90 min
   • Google File System (GFS) and MapReduce whitepapers
   • Doug Cutting and Apache Nutch origins
   • Commodity hardware philosophy
3. Hadoop architecture overview90 min
   • Master-slave topology pattern
   • Separation of storage (HDFS) and compute (YARN)
   • Daemon roles and cluster boundaries
4. Distributed storage and processing concepts90 min
   • Data locality principle
   • Batch processing paradigm over random I/O
   • Fault tolerance through redundancy
5. Setting up your Hadoop environment2 hrs
   • Java JDK compatibility matrices
   • SSH key generation for passwordless cluster access
   • hadoop-env.sh and environmental variables

Hands-on project: Configure a local single-node Hadoop pseudo-distributed environment using core-site.xml and hdfs-site.xml, verifying NameNode and DataNode daemons via port 9870.

You can manage block-level file distribution, diagnose under-replicated data, and execute HDFS shell commands to manipulate petabyte-scale datasets.

1. HDFS architecture: NameNode and DataNodes3 hrs
   • NameNode memory limitations and metadata overhead
   • EditLog and FsImage persistence
   • DataNode heartbeat mechanisms and block reporting
2. Blocks and replication3 hrs
   • Default block size sizing (128MB/256MB trade-offs)
   • Rack awareness configuration and placement policy
   • Client write pipeline and ACK packets
3. Reading and writing data3 hrs
   • Distributed file read and write sequence diagrams
   • Client-NameNode interaction overhead
   • Direct socket streaming to DataNodes
4. Fault tolerance and high availability3 hrs
   • Active-Standby NameNode configuration
   • ZooKeeper Failover Controller (ZKFC)
   • Quorum Journal Manager (QJM) state synchronization
5. HDFS commands and administration4 hrs
   • hdfs dfs -ls, -mkdir, -put, -get, and -cat
   • Safe mode management and quota enforcement
   • hdfs dfsadmin report and balancing tools

Hands-on project: Write a shell automation script that ingests raw log files into HDFS, inspects block distribution across DataNodes, and triggers fsck to detect missing blocks.

You can design, compile, and execute custom Java MapReduce jobs to parse raw telemetry logs and manage the shuffle phase efficiently.

1. The MapReduce programming model3 hrs
   • Functional primitives: map, filter, and reduce
   • Key-value pair serialization requirements
   • Writable comparable interfaces
2. Mappers, reducers and the data flow3 hrs
   • RecordReader and InputFormat parsing
   • Mapper output collection and serialization buffers
   • Reducer aggregation and OutputFormat writing
3. Shuffle, sort, combiners and partitioners4 hrs
   • Spill-to-disk mechanisms during shuffle
   • In-memory and combiner pre-aggregation
   • HashPartitioner vs custom partitioner logic
4. Writing and running MapReduce jobs3 hrs
   • Job configuration via JobConf and ToolRunner
   • Fat jar packaging with Maven assembly
   • YARN job submission and log aggregation
5. Strengths, limits and when to move beyond it3 hrs
   • Disk I/O bottlenecks in multi-stage pipelines
   • Iterative algorithm inefficiencies
   • Transition criteria from MapReduce to in-memory engines

Hands-on project: Build a Maven-based Java MapReduce job that aggregates daily clickstream traffic by IP address, implementing a custom partitioner to prevent reducer data skew.

You can configure YARN container memory limits, allocate capacity schedulers, and monitor cluster resource utilization under multi-tenant workloads.

1. Why YARN exists90 min
   • MapReduce 1.0 JobTracker scalability bottlenecks
   • Decoupling resource management from execution models
   • Generic cluster scheduler architecture
2. ResourceManager and NodeManagers90 min
   • ResourceManager scheduling heartbeats
   • NodeManager container lifecycle management
   • Container resource isolation via Linux cgroups
3. Application scheduling90 min
   • ApplicationMaster negotiation protocol
   • Container allocation requests and heartbeats
   • Node labels and node-restricted placement
4. Running multiple workloads90 min
   • Coexisting MapReduce, Spark, and Hive on YARN
   • Dynamic allocation tuning parameters
   • Container preemption behavior
5. Capacity and fair scheduling2 hrs
   • capacity-scheduler.xml configuration parameters
   • Hierarchical queues and weight guarantees
   • Fair scheduler preemption pools

Hands-on project: Configure YARN capacity scheduler queues with isolated resource allocations for batch vs ad-hoc workloads, and enforce memory-over-allocation safeguards.

You can create external and managed Hive tables, partition large datasets by date, and execute optimized HiveQL queries over HDFS data.

1. Hive on top of Hadoop90 min
   • SQL abstraction layer over MapReduce/Tez execution
   • Metastore architecture (Derby vs MySQL backend)
   • SerDe (Serializer-Deserializer) framework
2. HiveQL basics90 min
   • Data definition language (DDL): CREATE, ALTER, DROP
   • Managed vs external tables trade-offs
   • Primitive and complex data types (STRUCT, MAP, ARRAY)
3. Tables, partitioning and queries2 hrs
   • Static and dynamic partitioning syntax
   • Bucket pruning and sort-merge joins
   • Vectorized query execution settings
4. Where Hive fits in a pipeline90 min
   • ETL processing workflows with Hive scripts
   • Serving BI queries from data lake storage
   • Integration with enterprise reporting layers
5. Hands-on querying90 min
   • Beeline CLI connection and execution
   • EXPLAIN plan analysis for query optimization
   • Handling corrupt records and bad data files

Hands-on project: Design a partitioned Hive data warehouse schema for web log analytics, loading raw text files into optimized ORC storage format with snappy compression.

You can design wide-column schemas, manage region server distributions, and perform high-throughput CRUD operations in Apache HBase.

1. NoSQL and wide-column stores90 min
   • CAP theorem trade-offs in distributed datastores
   • Column families vs relational row structures
   • Sparse matrix storage model
2. HBase architecture90 min
   • HMaster and RegionServer daemon roles
   • HFiles, MemStores, and Write-Ahead Logs (WAL)
   • ZooKeeper coordination for cluster state
3. CRUD operations2 hrs
   • HBase shell commands: create, put, get, scan, delete
   • Row key design patterns and lexicographical sorting
   • Versioning and cell timestamp retrieval
4. HBase versus HDFS and Hive90 min
   • Random read/write vs sequential batch scanning
   • Latency profiles of wide-column stores
   • When to choose HBase over HDFS raw files
5. When to use HBase90 min
   • Real-time serving layers for analytical pipelines
   • Time-series data storage patterns
   • Anti-patterns: secondary indexing challenges

Hands-on project: Create an HBase table for user session tracking, selecting a composite row key to prevent region server hotspots during high-concurrency writes.

You can automate incremental data imports from relational databases into HDFS and stream live log events using ingestion agents.

1. Moving data with Sqoop90 min
   • JDBC driver configuration and connectivity
   • Parallel map-task allocation during imports
   • Password file security best practices
2. Importing from and exporting to RDBMS2 hrs
   • sqoop import and sqoop export syntax
   • Incremental imports using append and lastmodified modes
   • Direct mode imports for high-speed transfers
3. Streaming logs with Flume90 min
   • Flume agent topology: Source, Channel, Sink
   • Spooling directory and netcat sources
   • HDFS sink roll policies and file naming
4. Ingestion patterns90 min
   • Batch vs streaming ingestion architectural patterns
   • Handling schema evolution during transfers
   • Idempotent ingestion pipelines
5. Building a data pipeline90 min
   • Combining Sqoop, HDFS, and Hive tables
   • Automating daily ingestion shell scripts
   • Error handling and failure notifications

Hands-on project: Configure an Apache Sqoop incremental import job to pull daily transaction records from PostgreSQL into HDFS, and verify data integrity post-transfer.

You can construct data flow transformations in Pig Latin and orchestrate multi-step dependency graphs using Oozie workflow coordinators.

1. Pig and Pig Latin data flows90 min
   • Relational data algebra on Hadoop
   • Pig Latin script execution modes (local vs MapReduce)
   • Loading and storing relational tuples
2. Transforming data with Pig90 min
   • FILTER, FOREACH, GROUP BY, and JOIN operators
   • UDF (User Defined Functions) in Java and Python
   • Debugging execution plans with EXPLAIN and DUMP
3. Scheduling workflows with Oozie2 hrs
   • workflow.xml and coordinator.xml specifications
   • Directed Acyclic Graph (DAG) node definitions
   • Control flow nodes: fork, join, decision, kill
4. Coordinating jobs90 min
   • Time-based and data-availability triggers
   • Parameter substitution and workflow properties
   • Oozie web console job monitoring
5. Putting the ecosystem together90 min
   • End-to-end orchestration architectures
   • Managing workflow failures and checkpoint recovery
   • Transitioning from Oozie to modern schedulers

Hands-on project: Write an Oozie XML workflow that sequentially executes a Sqoop import, a MapReduce cleaning job, and a Hive partition load with automated retry logic.

You can monitor cluster health metrics, secure HDFS permissions with ACLs, and provision managed Hadoop clusters on cloud infrastructure.

1. Cluster setup and configuration basics90 min
   • Multi-node cluster networking and hostname resolution
   • Master-slave SSH trust configurations
   • XML configuration tuning across nodes
2. Monitoring and troubleshooting90 min
   • NameNode and YARNResourceManager web UI inspection
   • Interpreting container error logs and stack traces
   • JVM heap tuning for Hadoop daemons
3. Security fundamentals2 hrs
   • POSIX permissions and HDFS Access Control Lists (ACLs)
   • Kerberos authentication overview for Hadoop
   • Encryption in transit and at rest
4. Running Hadoop on the cloud (EMR)90 min
   • AWS EMR architecture and ephemeral clusters
   • S3 as a replacement or extension for HDFS
   • Cost optimization for cloud big data jobs
5. How Spark fits on Hadoop90 min
   • Apache Spark execution architecture on YARN
   • RDD caching vs HDFS disk persistence
   • Modernizing legacy Hadoop clusters with Spark

Hands-on project: Deploy an Apache Hadoop cluster on AWS EMR via Terraform, configure IAM security boundaries, and monitor cluster performance using Ganglia metrics.

You can replace a MapReduce job with a Spark job and explain why it runs faster.

1. Why Spark displaced MapReduce90 min
   • In-memory stages
   • DAG vs map-reduce pairs
   • When MapReduce still fits
2. RDDs, DataFrames and the Catalyst optimizer90 min
   • Lazy evaluation
   • Predicate pushdown
   • explain() plans
3. Submitting and sizing on YARN90 min
   • spark-submit cluster mode
   • executor-memory and cores
   • Dynamic allocation
4. Shuffles, skew and spilling90 min
   • Partition count tuning
   • Detecting skewed keys
   • Broadcast joins

Hands-on project: Rewrite a supplied two-stage MapReduce job as a Spark application, submit it to YARN in cluster mode with tuned executor memory and cores, and show the stage timeline and shuffle volume in the Spark History Server.

You can bring up a Kerberised cluster and restrict access to a dataset by column.

1. Kerberos in a Hadoop cluster90 min
   • KDC, realms and principals
   • Keytabs and kinit
   • Service vs user principals
2. Authorisation and HDFS permissions90 min
   • POSIX and ACLs
   • Ranger policy model
   • Doas and impersonation
3. Encryption at rest and in transit90 min
   • Encryption zones and the KMS
   • RPC and TLS wire encryption
   • Key rotation
4. Audit and compliance evidence90 min
   • Audit log destinations
   • Retention requirements
   • Producing an access report

Hands-on project: Kerberise a test cluster, create service principals and keytabs for HDFS and YARN, enable an HDFS encryption zone for a sensitive directory, and add a Ranger policy that masks one column for a non-privileged role.

You can diagnose a failed or slow job from the logs and counters rather than guessing.

1. Capacity planning and schedulers90 min
   • Capacity vs Fair Scheduler
   • Queues and preemption
   • Sizing from workload
2. Reading logs when a job dies90 min
   • yarn logs -applicationId
   • Container OOM kills
   • Counters as evidence
3. Common production failure modes90 min
   • Small-files problem
   • NameNode heap pressure
   • Straggler tasks
4. Monitoring, upgrades and recovery90 min
   • Ambari or Cloudera Manager metrics
   • Rolling restarts
   • NameNode HA failover

Hands-on project: Given a job that fails intermittently, use YARN application logs and HDFS metrics to identify the cause, correct the container memory configuration, and write a short post-incident note naming the failure mode and the fix.

You can design, deploy, and present an end-to-end production data pipeline incorporating ingestion, storage, processing, and query layers across the Hadoop ecosystem.

1. Ingest data into HDFS90 min
   • Designing ingestion schemas for raw incoming data
   • Automating source database extractions
   • Validating file integrity and block placement
2. Process it across the Hadoop stack90 min
   • Executing cleaning and aggregation jobs
   • Managing intermediate shuffle and spill states
   • Verifying YARN resource allocation efficiency
3. Query and analyse the results2 hrs
   • Building Hive data warehouse schemas over output files
   • Executing analytical queries and checking execution plans
   • Exporting processed results for reporting consumers
4. Document and present the pipeline90 min
   • Writing production-grade README and architecture diagrams
   • Documenting operational runbooks and failure recovery
   • Defending system design decisions in code reviews
5. Portfolio, resume, mock interviews and career prep90 min
   • Big data engineering interview question patterns
   • System design walkthroughs for distributed storage
   • Resume positioning for Hadoop and Data Engineer roles

Hands-on project: Deliver a complete big data capstone repository that ingests simulated financial telemetry via Sqoop into HDFS, processes logs via MapReduce, stores structured data in Hive, and undergoes a mock engineering review.