Projects

The core design was to decompose the task into two questions: what object is intended, and what grasp is physically executable. I used the VLM and SAM to convert open-ended language into a target-specific spatial prior, then restricted 3D inference to the masked point cloud so GraspNet reasons only over the commanded object instead of the full cluttered scene. The execution layer follows the same principle of controlled decomposition: candidate grasps are filtered by collision and approach geometry, ranked with object-centric priors, transformed from the camera frame into the robot world frame, and executed as a staged pre-grasp, grasp, lift, and place routine. This project is a language-to-action pipeline in which each module solves a well-defined subproblem and could be improved independently in future.

The key idea was to reduce the search space before learning: I first estimated a stable ROI for each fixed camera from the bright-band geometry, then built a reviewed pseudo-label batch and trained an ROI-only detector. During deployment, the system uses detector-first inference and falls back to heuristics when confidence is low, so the output is both robust and auditable.