Manipulation of Polymorphic Objects Using Two Robotic Arms through CNN Networks

Robinson Jimenez, Andres Jimenez, John Anzola


This article presents an interaction system for two 5 DOF (Degrees of Freedom) manipulators with 3-finger grippers, which will be used to grab and displace up to 10 polymorphic objects shaped as pentominoes, inside a VRML (Virtual Reality Modeling Language) environment, by performing element detection and classification using an R-CNN (Region Proposal Convolutional Neural Network), and point detection and gripping orientation using a DAG-CNN (Directed Acyclic Graph-Convolutional Neural Network). It was analyzed the feasibility or not of a grasp is determined depending on how the geometry of an element fits the free space between the gripper fingers. A database was created to be used as training data with each of the grasp positions for the polyshapes, so the network training can be focused on finding the desired grasp positions, enabling any other grasp found to be considered a feasible grasp, and eliminating the need to find additional better grasp points, changing the shape, inclination and angle of rotation. Under varying test conditions, the test successfully achieved gripping of each object with one manipulator and passing it to the second manipulator as part of the grouping process, in the opposite end of the work area, using an R-CNN and a DAG-CNN, with an accuracy of 95.5% and 98.8%, respectively, and performing a geometric analysis of the objects to determine the displacement and rotation required by the gripper for each individual grip.


robotic arms; R-CNN; DAG-CNN; 3-finger gripper; polymorphic objects; grip detection; robot interaction.

Full Text:



R. J. Moreno, P. C. U. Murillo, R. D. H. Beleño, Algorithm for Tool Grasp Detection, In: Colombia, International Review Of Mechanical Engineering ISSN: 1970-8734 v.12, fasc.1, p.1-8, DOI:, (2018).

A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, In Advances in neural information processing systems, pp. 1097-1105, (2012).

S. Yang, D. Ramanan, Multi-scale recognition with DAG-CNNs, In Computer Vision (ICCV), 2015 IEEE International Conference on, p. 1215-1223, (2015).

C. L. Zitnick, P. Dollár, Edge boxes: Locating object proposals from edges, European Conference on Computer Vision, Springer, Cham, p. 391-405. (2014).

R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, (2014).

I. Lenz, H. lee, A. Saxena, Deep learning for detecting robotic grasps, The International Journal of Robotics Research, vol. 34, no 4-5, p. 705-724. Doi: 10.1177/0278364914549607, (2015).

J. Redmon, A. Angelova, Real-time grasp detection using convolutional neural networks, En Robotics and Automation (ICRA), 2015 IEEE International Conference on, p. 1316-1322. Doi: 10.1109/ICRA.2015.7139361, (2015).

S. Kumra, C. Kanan, Robotic grasp detection using deep convolutional neural networks, arXiv preprint arXiv: 1611.08036, (2016).

Z. Wang, Z. Li, B. Wang, H. Liu, Robot grasp detection using multimodal deep convolutional neural networks, Advances in Mechanical Engineering, vol. 8, no 9, p. 1687814016668077. Doi:, (2016).

Cornell grasping dataset, data/ data.php, accessed: 2013-09-01.

N. Chen, S. Urban, C. Osendorfer, J. Bayer, P. V. D. Smagt, Estimating finger grip force from an image of the hand using convolutional neural networks and gaussian processes, In Proc. ICRA, (2014).

M. Meier, F Patzelt, R. Haschke, H. J. Ritter, Tactile convolutional networks for online slip and rotation detection, In International Conference on Artificial Neural Networks, Springer, Cham, pp. 12-19, (2016).

S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, D. Quillen, Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection, The International Journal of Robotics Research, vol. 37, no 4-5, p. 421-436.

Doi: 10.1177/0278364917710318, (2018).

A. Jin, S. Yeung, J. Jopling, J. Krause, D. Azagury, A. Milstein, L. Fei-Fei, Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks. arXiv preprint arXiv:1802.08774, (2018).

J. O. P. Arenas, R. J. Moreno, P. C. U. Murillo, Hand Gesture Recognition by Means of Region-Based Convolutional Neural Networks, Contemporary Engineering Sciences, vol. 10, no. 27, pp. 1329-1342, (2017).

J. Wang, J. D. MacKenzie, R. Ramachandran, D. Z. Chen, A deep learning approach for semantic segmentation in histology tissue images, In International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, pp. 176-184, (2016).

Y. F. Zheng, J. Y. S. LUH, Optimal load distribution for two industrial robots handling a single object, Journal of Dynamic Systems, Measurement, and Control, vol. 111, no 2, p. 232-237, (1989).

J. Y. S. LUH, Y. F. Zheng, Constrained relations between two coordinated industrial robots for motion control, The International journal of robotics research, vol. 6, no 3, p. 60-70, (1987).

D. Giardino, M. Matta, F. Silvestri, S. Spanò, V. Trobiani. FPGA implementation of hand-written number recognition based on CNN, In: International Journal on Advanced Science, Engineering and Information Technology, v. 9, fasc.1, pp. 167-171, (2019).



  • There are currently no refbacks.

Published by INSIGHT - Indonesian Society for Knowledge and Human Development