Applied Robotics and AI

You can establish the foundational architecture required to merge mechanical actuators with software-driven decision loops.

1. Recap of the sense–think–act loop2 hrs
   • Sensor telemetry sampling
   • State estimation feedback
   • Actuator command latency
2. What 'applied AI' means for robots2 hrs
   • Deterministic vs probabilistic control
   • Inference time budgets
   • Edge compute constraints
3. The architecture of an intelligent robot2 hrs
   • Hardware abstraction layers
   • Real-time operating systems
   • Inter-process communication topologies
4. Tools and environment setup2 hrs
   • Python virtual environments
   • Ubuntu LTS terminal workflows
   • Git repository initialization
5. Course project overview2 hrs
   • Capstone requirements breakdown
   • Simulation environment targets
   • Hardware validation checkpoints

Hands-on project: Map out the complete sense-think-act architecture for an autonomous mobile robot on a hardware node diagram.

You can manipulate sensor matrices and script deterministic robot behaviours using modern Python data libraries.

1. Python essentials for robotics2 hrs
   • Asynchronous event loops
   • Memory management in long-running scripts
   • Type hints and data classes
2. NumPy for sensor and matrix data2 hrs
   • Array slicing and vectorization
   • Homogeneous coordinate transformations
   • Matrix multiplication for rotations
3. Working with sensor streams2 hrs
   • Serial port data ingestion with PySerial
   • JSON and binary protocol parsing
   • Buffer overflow mitigation
4. Visualising robot data2 hrs
   • Matplotlib real-time plotting
   • Logging telemetry to disk
   • Headless debugging techniques
5. Scripting robot behaviour2 hrs
   • State machine implementation
   • Exception handling for hardware dropouts
   • Watchdog timer patterns

Hands-on project: Write a Python script that parses raw CSV telemetry logs from an IMU and computes rolling acceleration averages using NumPy.

You can combine noisy hardware feeds from IMUs, encoders, and lidar into a coherent environmental state estimate.

1. Advanced sensors and encoders2 hrs
   • Quadrature encoder tick counting
   • Wheel slip calculation
   • Analog-to-digital conversion
2. IMUs and motion sensing2 hrs
   • Accelerometer gravity compensation
   • Gyroscope bias drift calibration
   • Orientation quaternion math
3. Lidar basics2 hrs
   • Time-of-flight distance measurement
   • Laser scan message parsing
   • Point cloud data structures
4. Combining sensors (fusion)2 hrs
   • Kalman filter state prediction
   • Measurement update steps
   • Covariance matrix tuning
5. Interpreting the environment2 hrs
   • Occupancy thresholding
   • Dynamic obstacle flagging
   • Sensor noise filtering

Hands-on project: Implement a complementary filter in Python that fuses gyroscope rates and accelerometer angles to stabilize pitch estimation.

You can design decoupled robotics software architectures using ROS nodes, topics, publishers, and subscribers.

1. ROS architecture4 hrs
   • Master node and roscore daemon
   • Client libraries (rospy / rclpy)
   • Workspace directory structure
2. Nodes, topics and messages4 hrs
   • Custom message definitions (.msg files)
   • Topic introspection with rostopic
   • Message serialization formats
3. Publishers and subscribers4 hrs
   • Writing publisher node scripts
   • Writing subscriber callback functions
   • Queue size and buffer management
4. Launch files and parameters4 hrs
   • XML and Python launch file syntax
   • Dynamic parameter server configuration
   • Multi-node orchestration
5. Visualising with RViz4 hrs
   • Displaying TF coordinate frames
   • Rendering laser scan point clouds
   • Custom marker displays

Hands-on project: Build a multi-node ROS Python package where one node publishes sensor telemetry and another subscribes to log processed velocity commands.

You can construct physics-accurate robot models in URDF and validate sensory behaviors safely inside Gazebo.

1. Why simulate first2 hrs
   • Hardware failure prevention
   • Rapid algorithmic iteration
   • Edge case scenario generation
2. Building a robot model (URDF)2 hrs
   • Links, joints, and inertial tensors
   • Collision mesh simplification
   • Xacro macro templating
3. Simulating sensors2 hrs
   • Gazebo GPU lidar plugins
   • Depth camera noise models
   • IMU sensor noise configuration
4. Testing behaviours safely2 hrs
   • Automated regression testing in simulation
   • Physics engine parameter tuning
   • Collision logging
5. Sim-to-real basics2 hrs
   • Domain randomization concepts
   • Calibration discrepancies
   • Transfer validation steps

Hands-on project: Create a complete URDF model of a differential-drive robot with simulated lidar and camera plugins, and spawn it in a Gazebo world.

You can process raw camera frames, perform geometric calibrations, and extract visual features using OpenCV.

1. Images as data4 hrs
   • RGB and grayscale color spaces
   • NumPy multi-dimensional image arrays
   • Pixel manipulation and ROI slicing
2. OpenCV essentials4 hrs
   • cv2.imread and video capture streams
   • Geometric transformations and warping
   • Thresholding and contour extraction
3. Filtering and edge detection4 hrs
   • Gaussian blurring and noise reduction
   • Canny edge detection parameters
   • Hough line and circle transforms
4. Feature detection4 hrs
   • ORB and FAST keypoint descriptors
   • Feature matching across frames
   • Homography matrix estimation
5. Camera calibration4 hrs
   • Chessboard pattern detection
   • Intrinsic camera matrix calculation
   • Distortion coefficient correction

Hands-on project: Build an OpenCV pipeline that undistorts camera feeds from a wide-angle lens and tracks coloured target markers at 30 FPS.

You can detect and track specific objects in real time across dynamic video feeds.

1. Classification vs detection2 hrs
   • Image classification metrics
   • Bounding box regression
   • Non-max suppression algorithms
2. Training a simple vision model2 hrs
   • Dataset annotation formats
   • Transfer learning setup
   • Loss function monitoring
3. Running detection on a robot2 hrs
   • ONNX runtime model loading
   • Inference optimization for edge CPUs
   • Asynchronous frame queueing
4. Object tracking2 hrs
   • Centroid tracking algorithms
   • Kalman filter association
   • Handling occlusion events
5. Evaluating accuracy2 hrs
   • Mean Average Precision (mAP)
   • Confusion matrix generation
   • False positive reduction

Hands-on project: Deploy a lightweight MobileNet-SSD detector on a video stream to classify and bound shipping containers with confidence scores.

You can train and optimize neural networks for edge computing devices running under strict power and memory bounds.

1. ML concepts for perception4 hrs
   • Supervised learning pipelines
   • Overfitting and regularization
   • Hyperparameter tuning workflows
2. Supervised learning refresher4 hrs
   • Scikit-learn regression models
   • Feature scaling techniques
   • Cross-validation strategies
3. Neural networks for vision4 hrs
   • Convolutional layers and pooling
   • Activation functions (ReLU, Sigmoid)
   • Backpropagation mechanics
4. Deploying models on edge devices4 hrs
   • TensorFlow Lite conversion
   • Hardware accelerator delegation
   • Memory footprint profiling
5. Real-world performance4 hrs
   • Latency vs accuracy trade-offs
   • Batch processing optimization
   • Thermal throttling mitigation

Hands-on project: Quantize a PyTorch perception model to 8-bit integer weights and benchmark inference speed on a Raspberry Pi target.

You can compute robot odometry, manage coordinate frames, and construct spatial maps using SLAM algorithms.

1. Odometry and localisation2 hrs
   • Wheel encoder dead reckoning
   • Particle filter Monte Carlo localization
   • Pose estimation error growth
2. Mapping the environment (SLAM basics)2 hrs
   • Simultaneous Localization and Mapping principles
   • Grid-based occupancy mapping
   • Loop closure detection
3. Coordinate frames and transforms2 hrs
   • The tf2 transformation tree
   • Broadcasting static and dynamic transforms
   • Lookup transform time buffers
4. The navigation stack overview2 hrs
   • Global and local costmaps
   • Inflation layer configuration
   • Recovery behaviors setup
5. Introduction to path planning2 hrs
   • Dijkstra and A* grid search
   • Dynamic Window Approach (DWA)
   • Trajectory generation

Hands-on project: Run the Cartographer SLAM node against recorded lidar data in Gazebo to generate a 2D occupancy grid map of an indoor environment.

You can synthesize perception data and path-planning algorithms to execute autonomous obstacle avoidance and goal navigation.

1. Obstacle avoidance2 hrs
   • Reactive control loops
   • Vector field histograms
   • Emergency stop triggers
2. Path-planning algorithms2 hrs
   • NavFn global planner tuning
   • Base local planner velocity smoothing
   • Costmap clearing operations
3. Decision-making logic2 hrs
   • Behavior trees in robotics
   • Actionlib client-server patterns
   • Mission state sequencing
4. Combining perception and action2 hrs
   • Visual servoing loops
   • Dynamic target intercept algorithms
   • Closed-loop control adjustments
5. Autonomous behaviours2 hrs
   • Patrol routing scripts
   • Docking and charging routines
   • Failure recovery sequences

Hands-on project: Configure the ROS navigation stack action server to drive a robot safely through a cluttered obstacle course to a target waypoint.

You can harden robotics software with rigorous integration testing, performance profiling, and safety monitoring.

1. Integrating vision, ML and control2 hrs
   • Thread safety across ROS nodes
   • Zero-copy message transport
   • Resource contention profiling
2. Debugging robot systems2 hrs
   • ROS bag recording and playback
   • RQt graph inspection tools
   • Log level filtering and tracing
3. Reliability and safety2 hrs
   • Hardware watchdog implementation
   • E-stop signal wiring and software hooks
   • Failsafe state transitions
4. Performance tuning2 hrs
   • CPU core affinity pinning
   • Network bandwidth optimization
   • Latency bottleneck identification
5. Documentation2 hrs
   • System architecture diagrams
   • API documentation with Sphinx
   • Deployment runbook creation

Hands-on project: Write unit tests for custom ROS nodes using pytest, profile CPU memory usage with top/htop, and containerize the stack in Docker.

You can take a robot off the bench safely and justify every fail-safe it carries.

1. Risk assessment for robots90 min
   • Hazard identification
   • Severity and likelihood
   • Functional safety concepts
2. Fail-safe by design90 min
   • Emergency stop circuits
   • Watchdog timers
   • Safe default states
3. Degrading rather than failing90 min
   • Sensor dropout handling
   • Velocity and torque limits
   • Geofencing
4. Operating outside the lab90 min
   • Pre-flight checklists
   • Logging for incidents
   • Maintenance and calibration

Hands-on project: Produce a risk assessment for the capstone robot identifying hazards and mitigations, implement an emergency stop and a watchdog that halts motion on lost communication, and demonstrate safe behaviour under sensor failure.

You can architect, build, and deploy an end-to-end intelligent robotic system capable of autonomous navigation and visual perception.

1. Designing an autonomous behaviour4 hrs
   • Mission requirements analysis
   • State machine architecture design
   • Safety interlock planning
2. Building and integrating4 hrs
   • Merging perception and localization
   • Connecting sensor nodes to planner
   • Bringing up hardware or simulation world
3. Testing in the real world4 hrs
   • Field test execution protocols
   • Edge case identification
   • Telemetry log analysis
4. Improving reliability4 hrs
   • Addressing sensor dropout failures
   • Tuning control loop parameters
   • Refactoring unstable nodes
5. Showcasing your robot4 hrs
   • Preparing code repositories for review
   • Recording demonstration runs
   • Technical architecture presentation

Hands-on project: Deploy the complete autonomous robot capstone in simulation, verify perception-driven navigation across randomized obstacles, and record a system validation audit.