Method for automatic object identification and robotic grasp point determination using machine vision

RELEVANCE of the study lies in the automation of manipulation operations involving complex-shaped objects in industrial production, which remains an important task in modern robotics. In many practical applications, objects are located on the working surface of a robotic system without overlapping but may have arbitrary orientations, which significantly complicates their automatic identification and the determination of grasp coordinates for a robotic manipulator. Although three-dimensional machine vision systems can solve this problem, they usually require RGB-D cameras and high-performance computing hardware, which significantly increases the cost and complexity of robotic systems. Therefore, the development of efficient methods capable of solving this task using only a monocular camera remains relevant.

THE PURPOSE of the study is to develop and experimentally investigate a method for automatic identification of complex-shaped objects and determination of their grasp coordinates by a robotic manipulator using a machine vision system equipped with a single monocular camera.

METHODS. Object recognition in digital images is performed using a convolutional neural network based on the YOLOv8 architecture, which allows detection of objects and estimation of their orientation on the working surface. The grasp coordinates are determined by analyzing the bounding box of the detected object and applying an experimentally determined offset of the grasp point relative to the geometric center of the object.

RESULTS. Experimental evaluation was carried out using chess pieces of different shapes and colors as test objects. The object detection confidence was not lower than 0.9, while the processing time of a single image was less than 56 ms, which ensures real-time operation.

CONCLUSIONS. The proposed method enables automatic identification of complex-shaped objects and stable determination of their grasp coordinates using only a single monocular camera without expensive three-dimensional sensors.

1. Schetcikov Yu.A., Kainova T.D., Chupakov M.V., Demidova K.A. Target recognition based on computer vision technologies for unmanned systems. Proceedings of Tula State University. Technical Sciences. 2023, Issue 7, pp. 628–629. DOI: 10.24412/2071-6168-2023-7-628-629.

2. Jocher G., Chaurasia A., Qiu J. YOLO by Ultralytics. Version 8.0.0. Ultralytics, 2023. Available at: https://docs.ultralytics.com

3. Redmon J., Divvala S., Girshick R., Farhadi A. You Only Look Once: Unified, Real-Time Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788.

4. Kaznacheeva A.A., Vlasenko O.M., Epov A.A. Control algorithm for a mechatronic product sorting station using a computer vision system. Engineering Journal of Don. 2025, No. 5.

5. Ershov M.S., Zhuravlev S.M. Application of artificial neural networks for monitoring stability parameters of industrial electrical systems. News of Higher Educational Institutions. Energy Sector Problems. 2025, Vol. 27, No. 5, pp. 41–52.

6. Politova T.O., Zagretdinov A.R., Sidorov M.V., Ziganshin Sh.G., Vankov Yu.V. Machine learning methods in the development of leak detection techniques. News of Higher Educational Institutions. Energy Sector Problems. 2025, Vol. 27, No. 2, pp. 177–186.

7. Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C.-Y., Berg A.C. SSD: Single Shot MultiBox Detector. European Conference on Computer Vision (ECCV), 2016, pp. 21–37.

8. Redmon J., Farhadi A. YOLOv3: An Incremental Improvement. arXiv preprint arXiv:1804.02767.

9. Bochkovskiy A., Wang C.-Y., Liao H.-Y.M. YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv preprint arXiv:2004.10934.

10. Redmon J., Farhadi A. YOLO9000: Better, Faster, Stronger. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

11. Wang C.-Y., Bochkovskiy A., Liao H.-Y.M. YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 7464–7475.

12. Bokov P.A., Kravchenya P.D. Experimental analysis of accuracy and performance of YOLO architectures for computer vision tasks. Software & Systems. 2020, No. 4, pp. 635–640.

13. Nebaba S.G., Markov N.G. Convolutional neural networks of the YOLO family for mobile computer vision systems. Computer Research and Modeling. 2024, No. 3, pp. 615–631.

14. Gladkov A.N., Bystrov L.Yu., Kuzmin E.V. Application of YOLO neural networks for detection of useful signals in eddy-current rail defectograms. Proceedings of the Institute for System Programming of the Russian Academy of Sciences. 2025, Vol. 37, No. 6, pp. 131–150. DOI: 10.15514/ISPRAS-2025-37(6)-25.

15. Shilin A.N., Bogale M.A., Konovalova L.A. Artificial neural network model for forecasting water volume in a reservoir. News of Higher Educational Institutions. Energy Sector Problems. 2024, Vol. 26, No. 5, pp. 104–117.

16. Zhang H., Lan X., Zhou X. RGB-D Based Robotic Grasp Detection: A Survey. Sensors. 2022, Vol. 22, No. 15, p. 5634.

17. Ma Y., Zhu W., Zhou Y. Automatic grasping control of a mobile robot based on monocular vision. The International Journal of Advanced Manufacturing Technology. 2022, Vol. 121, pp. 1785–1798. DOI: 10.1007/s00170-022-09438-z.

18. Andreev V.P. Determining the shape of an obstacle using a mobile robot sensor system based on GP2Y0A infrared distance sensors (Sharp). News of Higher Educational Institutions. Energy Sector Problems. 2024, Vol. 26, No. 1, pp. 195–207.

19. Rubleva E.A., Mudrich A.B. Determining the position of a robot manipulator tool using a camera and machine vision. International Research Journal. 2024.

20. Mahler J., Liang X., Niyaz S., Peters D., Rich A., Lee J., Gonzalez A., Goldberg K. Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. Robotics: Science and Systems (RSS), 2017.

21. Polivanov A.Yu., Dabul Yu. Method for determining coordinates of welding joints in a robot vision system for arc welding. STANKIN University Bulletin. 2025, No.1 (72), pp. 16–25.

22. Polivanov A.Yu., Ivanov Y.V. Calibration of the video sensor of the technical vision system of an industrial robot for laser welding. Messenger of MSUT “STANKIN”. 2019. No.49, pp.119-126.