Linux for Big Data

You can spin up an isolated virtual environment and navigate complex directory trees using core shell commands without relying on a graphical user interface.

1. What Linux is: the kernel, distributions and open source90 min
   • Linux kernel vs GNU userland
   • Distributions: Debian, Ubuntu, RHEL and Alpine
   • Open source licensing and the command-line philosophy
2. Installing Linux: virtual machines, WSL and the cloud90 min
   • VirtualBox and KVM hypervisors
   • Windows Subsystem for Linux (WSL2)
   • Launching EC2 or cloud VM instances via SSH
3. The shell, the terminal and your first commands90 min
   • Bash and Zsh shell environments
   • Standard input, output and exit codes ($?)
   • echo, pwd, history and clear commands
4. Navigating files and directories90 min
   • Absolute vs relative paths
   • cd, ls with flags (-la, -lh), mkdir and rmdir
   • Using tab completion and command history shortcuts
5. Getting help: man, --help and info90 min
   • Reading man page sections (1 through 8)
   • Using command --help and type flags
   • Navigating built-in search with less inside man

Hands-on project: Provision a local Ubuntu container or virtual machine via WSL, navigate the root directory structure, and document the kernel version, CPU architecture, and available memory using system utility flags.

You can inspect, manipulate, link, and query files across the filesystem hierarchy with surgical precision.

1. The Linux Filesystem Hierarchy (FHS)90 min
   • /etc, /var, /usr, /home and /proc directories
   • Mount points and virtual filesystems
   • Temporary storage in /tmp and /var/tmp
2. Creating, copying, moving and removing files90 min
   • touch, cp (-r, -p), mv and rm (-rf safety)
   • Batch file manipulation with wildcards (*, ?, [])
   • Safe deletion practices in production environments
3. Viewing and editing files: cat, less, nano and vim basics90 min
   • cat, head, tail -f for live file inspection
   • Paging through large logs with less
   • Vim modal editing: insert mode, normal mode, saving and quitting
4. Hard links and symbolic links90 min
   • Inode architecture and hard link mechanics
   • Creating symbolic (soft) links with ln -s
   • Debugging broken symlinks in application paths
5. Finding files: find, locate and which90 min
   • find with -name, -type, -size, and -mtime filters
   • Executing inline commands via find -exec
   • Searching executable binaries with which and type

Hands-on project: Create a nested directory tree mirroring a production log archive, link configuration files across directories using symbolic and hard links, and locate stale files using complex find criteria.

You can configure multi-user access control lists, enforce file permissions, and secure administrative privilege escalation.

1. Users, groups and the /etc/passwd and /etc/group files90 min
   • useradd, usermod, userdel and passwd commands
   • Parsing /etc/passwd, /etc/shadow and /etc/group
   • UIDs, GIDs and system vs regular accounts
2. File permissions: read, write and execute (chmod)90 min
   • Symbolic (u+rwx) vs numeric (755, 644) modes
   • Permissions on files versus directories
   • Default permissions masking with umask
3. Ownership with chown and chgrp90 min
   • Changing user and group ownership with chown
   • Recursive ownership updates across directory trees
   • Troubleshooting permission denied errors in production
4. Special permissions: setuid, setgid and the sticky bit90 min
   • SUID and SGID security implications
   • Setting the sticky bit on shared directories like /tmp
   • Numerical representation of special permission bits
5. sudo and privilege escalation90 min
   • Configuring /etc/sudoers with visudo
   • Granting specific command privileges to user groups
   • Auditing administrative privilege usage

Hands-on project: Set up a shared directory for a data engineering team, assign primary and secondary groups, configure sticky bits for secure scratch space, and restrict root access using sudoers configuration.

You can extract, transform, and filter gigabytes of log or data streams using standard Unix text-processing utilities.

1. Standard input, output and error; redirection90 min
   • File descriptors (stdin 0, stdout 1, stderr 2)
   • Redirection operators (> , >> , 2>&1, <)
   • Discarding output via /dev/null
2. Pipes and chaining commands together90 min
   • Passing output between commands using the pipe (|) operator
   • Building modular command-line pipelines
   • Handling buffer limits and stream processing
3. grep and regular expressions90 min
   • Basic vs Extended Regular Expressions (BRE/ERE)
   • Using grep, egrep, and rgrep for pattern matching
   • Context flags (-A, -B, -C, -v, -i)
4. Transforming text with sed and awk90 min
   • Stream editing with sed substitute (s/foo/bar/g)
   • Pattern scanning and field processing with awk
   • Defining custom field separators (-F) and print blocks
5. sort, cut, uniq, wc and tr90 min
   • Extracting columns with cut and delimiters
   • Counting frequencies with sort and uniq -c
   • Character translation and deletion with tr and wc

Hands-on project: Build an analytical pipeline using pipes, grep, sed, and awk to parse multi-gigabyte server access logs, extract error rates, and generate a formatted summary report.

You can isolate resource-hungry processes, manage background jobs, and diagnose system performance bottlenecks in real time.

1. Processes, PIDs and the process tree90 min
   • Process lifecycle, parent/child relationships and init (PID 1)
   • Inspecting process trees with pstree
   • Foreground and background execution (&, bg, fg, jobs)
2. Foreground, background and job control90 min
   • Suspending jobs with Ctrl+Z
   • Resuming background tasks safely
   • Detaching processes with nohup and disown
3. ps, top, htop and kill90 min
   • ps aux and custom output formatting (-o)
   • Interactive system monitoring with top and htop
   • Sending signals with kill, pkill and killall
4. Signals and process priority (nice and renice)90 min
   • Understanding SIGINT, SIGTERM, SIGKILL and SIGHUP
   • CPU scheduling priority with nice and renice values (-20 to 19)
   • Checking I/O pressure and load averages
5. Monitoring CPU, memory and disk usage90 min
   • Memory tracking with free -h and /proc/meminfo
   • Disk I/O monitoring with iostat and vmstat
   • Identifying swap usage and memory leaks

Hands-on project: Identify a runaway background process consuming excessive CPU, inspect its process tree and open file descriptors, adjust its priority, and terminate it cleanly using appropriate signals.

You can install, upgrade, and manage software packages, repositories, and third-party runtimes across Debian and Red Hat distributions.

1. Package managers: APT (Debian/Ubuntu) and DNF/YUM (RHEL)90 min
   • Package manager architecture and metadata caching
   • APT commands: update, upgrade, install, purge
   • DNF/YUM equivalents and module streams
2. Installing, updating and removing packages90 min
   • Managing local .deb and .rpm installer files
   • Handling broken dependencies and held packages
   • Autoremoving orphaned libraries and cache cleaning
3. Repositories and dependency management90 min
   • Managing sources in /etc/apt/sources.list and PPA repositories
   • Importing and trusting GPG signature keys
   • Pinning package versions for system stability
4. Compiling software from source (overview)90 min
   • The configure, make, make install workflow
   • Managing build dependencies and header files
   • Tracking installed files without package managers
5. Snap and Flatpak90 min
   • Containerised application packaging models
   • Installing and managing Snaps and Flatpaks
   • Trade-offs between native packages and universal formats

Hands-on project: Add an external software repository to an Ubuntu server, resolve dependency conflicts during package installation, and verify software authenticity using GPG keys.

You can write robust, production-grade Bash scripts complete with error handling, argument parsing, and automated logging.

1. Writing and running your first Bash scripts120 min
   • Shebang lines (#!/bin/bash) and execution permissions
   • Running scripts with bash script.sh vs ./script.sh
   • Debugging scripts with set -x and bash -v
2. Variables, arguments and reading user input120 min
   • Defining, exporting and referencing scalar variables
   • Positional parameters ($1, $2, $@, $#, $$)
   • Interactive input collection with read and default values
3. Conditionals: if, case and test120 min
   • Integer, string and file test operators ([ ], [[ ]])
   • Branching logic with if/elif/else blocks
   • Pattern matching with case statements
4. Loops: for, while and until120 min
   • Iterating over lists and ranges with for loops
   • Processing streams line-by-line with while loops
   • Control statements: break, continue and exit codes
5. Functions, exit codes and practical automation scripts120 min
   • Defining and calling modular Bash functions
   • Returning values and setting strict error trapping (set -euo pipefail)
   • Writing production-ready error handling and logging wrappers

Hands-on project: Develop an automated data-backup and log-rotation script that validates input directories, compresses archives with timestamping, checks exit codes, and logs execution output securely.

You can configure network interfaces, diagnose TCP/IP connectivity issues, establish secure remote tunnels, and harden firewall rules.

1. Networking basics: IP addresses, ports and DNS90 min
   • IPv4/IPv6 addressing, subnets and CIDR notation
   • TCP and UDP protocol fundamentals
   • DNS resolution, /etc/hosts and name servers
2. Configuring interfaces and checking connectivity90 min
   • ip route, ip addr and network interface management
   • Testing routing with traceroute and mtr
   • Testing port reachability with nc (netcat)
3. ss, netstat, ping, curl and wget90 min
   • Inspecting listening sockets and open ports with ss
   • HTTP endpoint testing and downloading with curl and wget
   • ICMP connectivity checks with ping
4. Secure remote access with SSH90 min
   • Generating Ed25519/RSA SSH key pairs with ssh-keygen
   • Copying keys securely using ssh-copy-id
   • Hardening /etc/ssh/sshd_config and tunneling
5. Firewalls: iptables and ufw90 min
   • Uncomplicated Firewall (UFW) syntax and status rules
   • Packet filtering chains and tables in iptables
   • Allowing and denying traffic by port, IP and interface

Hands-on project: Configure a secure SSH daemon with key-only authentication, disable root logins, set up custom port forwarding, and lock down incoming traffic using UFW firewall rules.

You can partition raw disks, configure persistent mount points, manage logical volumes, and troubleshoot system boot sequences.

1. Disks, partitions and filesystems90 min
   • Identifying block devices in /dev (sda, nvme0n1)
   • Partition tables: MBR vs GPT partitioning with parted and fdisk
   • Formatting filesystems with mkfs.ext4 and mkfs.xfs
2. Mounting drives and configuring /etc/fstab90 min
   • Manual mounting and unmounting with mount and umount
   • Persistent mounts via UUID in /etc/fstab
   • Diagnosing mount failures and emergency rescue mode
3. Logical Volume Management (LVM) overview90 min
   • Physical Volumes (PV), Volume Groups (VG) and Logical Volumes (LV)
   • Creating and extending logical volumes dynamically
   • Resizing filesystems online without data loss
4. The boot process and GRUB90 min
   • BIOS/UEFI boot stages and master boot records
   • GRUB2 bootloader configuration and kernel parameters
   • Recovering a broken boot loader from a live USB environment
5. An introduction to systemd and services90 min
   • Systemd target states and runlevels
   • The boot dependency graph and startup profiling
   • Analyzing boot performance with systemd-analyze

Hands-on project: Partition a raw block device, configure Logical Volume Management (LVM) with a resizable data pool, mount the filesystem persistently in /etc/fstab, and inspect GRUB boot parameters.

You can configure system daemons, automate scheduled tasks, analyze centralized logs, and harden server security against common attack vectors.

1. Managing services and daemons with systemd (systemctl)90 min
   • Controlling services: start, stop, restart, enable, disable
   • Writing custom systemd unit files in /etc/systemd/system/
   • Configuring automatic restart policies and resource limits
2. Scheduling tasks with cron and at90 min
   • Cron expression syntax (minute, hour, day, month, weekday)
   • Managing user crontabs vs system crontabs (/etc/crontab)
   • One-off job scheduling with the at command
3. Logs, journald and log rotation90 min
   • Querying system logs with journalctl (time, unit, priority filters)
   • Traditional log files in /var/log/syslog and /var/log/auth.log
   • Configuring rotation policies with logrotate configuration files
4. User and system security hardening90 min
   • Disabling unused network services and ports
   • Configuring password aging policies and account lockouts
   • Security auditing with Lynis and basic SELinux/AppArmor overview
5. Backups, recovery and troubleshooting90 min
   • Archive creation and extraction with tar, gzip and rsync
   • Incremental vs full backup strategies
   • Troubleshooting disk space exhaustion and inode starvation

Hands-on project: Create a custom systemd service for a background application, schedule automated backups via cron, configure log rotation policies with logrotate, and audit system security.

You can run cluster daemons as supervised services and recover a node that fails to start.

1. Units and dependency ordering90 min
   • After and Requires on mounts
   • Restart backoff
   • systemctl status forensics
2. Resource limits for JVM daemons90 min
   • MemoryMax vs heap
   • LimitNOFILE for many sockets
   • OOM score adjustment
3. Journals and log rotation90 min
   • journalctl -u since
   • logrotate for app logs
   • Disk filling from logs
4. Recovering a node90 min
   • Failed mount recovery
   • fsck on a data volume
   • Rejoining the cluster

Hands-on project: Package a data-node daemon as a systemd unit with restart backoff and a memory cap, deliberately break its mount, then recover the node and explain the failure using journalctl and dmesg.

You can tune a Linux host for sequential big-data I/O and show the throughput gain.

1. Where big-data hosts actually hurt90 min
   • Sequential vs random I/O
   • Page cache behaviour
   • Network saturation
2. Disk and filesystem tuning90 min
   • noatime and mount options
   • Read-ahead settings
   • I/O scheduler choice
3. Memory, swap and NUMA90 min
   • vm.swappiness
   • Transparent huge pages
   • numactl placement
4. Benchmarking honestly90 min
   • fio job files
   • Warm vs cold cache runs
   • Reporting percentiles

Hands-on project: Benchmark disk throughput with fio before and after tuning the I/O scheduler, read-ahead and mount options on a data volume, raise file-descriptor and process limits for the daemon, and report the measured change.

You can independently provision, configure, secure, and maintain a production-grade Linux server from scratch.

1. Set up and secure a Linux server on the cloud or a VM90 min
   • Provisioning a fresh cloud VM or local virtual machine
   • Enforcing SSH key-only authentication and disabling root login
   • Configuring hostnames, timezones and NTP synchronization
2. Manage users, services, networking and storage90 min
   • Creating operational user accounts with restricted sudo access
   • Configuring LVM storage pools and persistent fstab mounts
   • Establishing firewall rules and testing network accessibility
3. Automate routine tasks with shell scripts and cron90 min
   • Writing automated maintenance and backup scripts
   • Registering cron jobs with proper error redirection
   • Testing end-to-end automation scripts in staging
4. Configure logging, monitoring and a firewall90 min
   • Setting up persistent systemd journal logging
   • Configuring log rotation for custom application logs
   • Verifying UFW firewall policies and socket listeners
5. Portfolio, GitHub, mock interviews and career prep90 min
   • Documenting server architecture and runbooks on GitHub
   • Preparing for Linux system administration interview questions
   • Mock technical reviews and troubleshooting assessments

Hands-on project: Deploy a cloud virtual machine, configure LVM storage, set up user groups and SSH hardening, deploy a custom systemd service with automated backups, and pass a comprehensive security and uptime audit.