Text-prompt Detection and Prompt Segmentation (Experimental)
This workflow uses natural-language prompts, boxes, points, or 2D regions generated from point clouds to guide detection and instance segmentation. It is useful for rapid evaluation of new categories or projects without a dedicated trained model. Validate accuracy, runtime environment, stability, and cycle time with production data before deployment.
Available operators
-
Text-prompt Detection locates objects from English words or sentences.
-
Fast Prompt Segmentation generates masks from boxes, points, or optional text prompts.
-
Universal Segmentation V3 performs open-vocabulary detection and segmentation from English prompts.
For production projects with strict repeatability, also compare dedicated Detection Model V1, Segmentation Model V1, or Detection and Segmentation.
Workflow 1: text detection followed by prompt segmentation
-
Apply an image ROI when required.
-
Use Text-prompt Detection with a concise English target description.
-
Use Split Detection Result to obtain prompt boxes or center points.
-
Provide the image and prompts to Fast Prompt Segmentation.
-
Filter by class, confidence, area, and overlap before point-cloud extraction or pose generation.
Start with a simple category name and add only visible distinguishing attributes. When multiple categories are requested, verify prompt separation and class mapping used by downstream filters.
Workflow 2: generate prompts from point-cloud boxes
Segment the camera-frame point cloud using ROI, clustering, height, or normals. Compute 3D boxes with Get Point-cloud Bounding Box, then project them with Convert Point-cloud Box to 2D Box. Use the resulting boxes as segmentation prompts.
The boxes must use the camera coordinate system associated with the selected camera parameters. If all boxes are shifted, inspect the image source, coordinate system, and calibration instead of compensating with excessive box expansion.
Workflow 3: generate prompts from point-cloud masks
Project each target cloud with Convert Point Cloud to Mask Image. Apply only the morphology needed to fill gaps, then convert masks with Generate Detection Results from Mask List. Split the detections and use their boxes or centers as segmentation prompts.
Use polygon masks when projection is complete, minimum-area rectangles when a stable box is sufficient, and direct projected points when checking raw projection quality.
Tuning and limitations
-
Prompt boxes usually provide stronger spatial constraints than a single point. Validate small targets when reducing inference image size.
-
Increase confidence to remove weak results and tune overlap thresholds for duplicates. Do not hide alignment errors by enlarging prompts.
-
Universal Segmentation V3 has a shorter pipeline; validate prompts, confidence, NMS overlap, and class separation independently.
-
Semantic relevance does not guarantee the exact industrial class boundary, direction, or size definition.
-
Image, point cloud, camera parameters, and hand-eye data must still belong to the same acquisition for robot picking.
-
Define handling for empty, excessive, or low-confidence results before sending output to a robot.