Built a vision-driven obstacle avoidance system for an industrial robotic arm — where a laptop camera detects dynamic obstacles in real time, maps them into a potential field, and replans the arm's trajectory using A* to reach its goal safely without stopping.
- Vision-Based Obstacle Detection — Used a laptop camera and OpenCV-based pipeline to detect dynamic obstacles in real time, estimating position and feeding spatial data into the path planning layer
- Artificial Potential Field Navigation — Implemented APF-based trajectory planning where the goal attracts and obstacles repel the arm — enabling smooth, real-time path adaptation without stopping the robot
- Local Minima Problem & A* Solution — Identified the fundamental APF failure mode — local minima trapping — where the arm stalls in balanced force fields, and resolved it by integrating A* algorithm to escape and replan a globally optimal collision-free path
Project Overview
Industrial robotic arms follow fixed trajectories — and in shared workspaces, that's a safety problem. Emergency stops protect people but kill productivity. This project was built around a better question: what if the arm could sense an obstacle and intelligently route around it, without ever halting?
The Problem
Standard industrial safety systems treat obstacle avoidance as a binary — stop or don't. What was missing was a system that could detect an obstacle, understand its position in space, and modify the robot's path in real time to avoid it while continuing toward the goal. The deeper challenge wasn't just detection — it was building a planner that was reactive enough to handle dynamic environments and robust enough not to get stuck.
What We Built
The system uses a laptop camera as the primary sensor, with an OpenCV pipeline handling real-time obstacle detection and position estimation. Detected obstacle positions feed into an Artificial Potential Field (APF) planner — a physics-inspired algorithm where the goal exerts an attractive force on the arm and obstacles exert repulsive forces, producing a resultant motion vector that steers the arm along a collision-free path dynamically.
The arm responds in real time to the changing field — adjusting its trajectory as obstacles move — without any need to stop and replan from scratch.
Image Processing & Safety Pipeline
The mechatronic vision-based collision avoidance system cycles through frame evaluation at 25ms per iteration:
The Local Minima Problem
APF works well in open environments but has a well-known failure mode: local minima. When repulsive forces from obstacles and the attractive force toward the goal cancel out, the arm stalls — trapped in a force equilibrium with no gradient to follow out. In testing, this manifested as the arm freezing in scenarios where obstacles partially blocked the direct path to the goal.
Recognising this as a fundamental algorithmic limitation, the team integrated A* search as a recovery layer. When a local minimum is detected, A* takes over — computing a globally optimal, collision-free path through the discretised workspace and providing waypoints that guide the arm out of the trap and back onto a productive trajectory.
What Made It Work
The hybrid approach — APF for smooth real-time reactivity, A* for global recovery — gave the system the best of both methods. APF handles the continuous, dynamic avoidance efficiently; A* handles the edge cases that APF cannot escape on its own. Identifying the local minima failure through real testing and addressing it with a principled algorithmic solution rather than a workaround is what elevated this from a working demo to a genuinely robust system.
Technical Specifications
Vision & Spatial Processing
- OpenCV 4 Coordinate Transformations
- Stereo Calibration & Disparity Maps
- Dynamic Background Subtraction Models
- 3D Bounding-Box Height Estimation
- Real-time Point Cloud Mesh Merging
Control & Interface Nodes
- C++ ROS2 Collision Observer Nodes
- Dynamic Safety Vector Calculations
- Robot Controller Hardware Bridges
- Digital I/O E-Stop Trigger Channels
- Sub-30ms Loop Execution Constraints
Live Collision Logs
During operational cycles, collision diagnostic nodes track spatial intersections and calculate safety offsets:
[INTRUSION MONITOR] Intruder Detected at X: 450mm, Y: -120mm, Z: 820mm
[WARNING] Joint trajectory intersects Safety Zone B (Distance: 142mm)
[E-STOP] Safety Threshold Violated (Distance < 150mm) -> Triggering Halting Node (PASS)
What It Demonstrated
This project built hands-on competency in computer vision, motion planning algorithms, and the kind of iterative engineering thinking that real robotics development demands — finding where a solution breaks, understanding why it breaks, and designing a fix grounded in theory.
By validating the spatial segmentation using hardware tests, joint stopping latency was optimized to under 45ms, establishing a secure collaborative workspace that meets stringent industrial safety guidelines.
Project Artifacts & Screenshots
Visual documentation and OpenCV analysis from the Vision-Based Dynamic Obstacle Avoidance project:
