Hiroshi Kasuga, Lisa Hakataya, Hiroki Yokohata, The University of Tokyo
Introduction
On August 26 and 27, students and enthusiasts gathered at JAMSTEC headquarters in Yokosuka, Japan, to participate in the Underwater Robot Convention in JAMSTEC 2023. The convention, hosted by NPO Japan Underwater Robot Network, serves as a place for participants to exchange technical ideas and make connections through the competition and presentations of self-made underwater robots. The overview of the convention can be found on the official website [1] (in Japanese), and the conventions in previous years are reported in [2], [3], [4], [5], and [6].
We, the authors, are master’s course students in Prof. Maki’s laboratory at the University of Tokyo. We participated in the AI-challenge division as team “MakiLabM1” [Figure 1]. Our main objective was to acquire basic knowledge and skills on underwater robotics in preparation for future research works. We also learned about teamwork, which is important in the development and operation of underwater robots.
Competition Rules
The AI-challenge division was established to encourage the introduction of AI (Artificial Intelligence) into underwater robotics. Robots were required to autonomously break balloons placed in the water tank. There were three types of balloons: red, yellow, and blue, and multiple balloons were set up for each. The height from the bottom of the water tank at which the balloons were placed, and the score obtained by breaking them, differed depending on their color [Table 1]. The arrangement of them was random. Figure 2 shows the overview of the setup of the water tank. The only tool allowed to break them was one thumbtack. Each team was given 4 minutes to perform, and the higher the score obtained, the higher the competition score. In addition to the competition score, the judges also evaluated the degree of autonomy of the robots and the presentations in the poster session, and the final ranking was determined by the total of these scores.
Table.1. Height and score of balloons
| Color | Height [m] | Score |
| Red | 0.3 | 30 |
| Yellow | 0.8 | 20 |
| Blue | 0.6 | -10 |
Strategy
Sebastian is an AUV that can move nimbly in surge, heave, and yaw directions. We implemented various algorithms in order to break more red balloons within the time limit and get a higher score. The following is a description of the algorithm implemented along with the competition flow. At the beginning of the competition, Sebastian is at the edge of the pool. At the signal to start the competition, Sebastian moves to the center of the pool where the balloons are densely packed. Once in the center of the pool, Sebastian uses the heave thruster to land on the bottom of the pool. Sebastian, with an altitude of 0 cm at this landing point, rises to an altitude of 30 cm, where the red balloons are located. This makes it easier to recognize the red balloons with the highest score. After surfacing to an altitude of 30 cm, Sebastian starts autonomous navigating and searching for balloons. Fig. 3 shows a conceptual diagram of the balloon search mode. Sebastian remembers the nearest balloon A and the nearest balloon B from balloon A. While approaching A, another balloon C is seen. If balloon C is farther away than balloon A and closer than balloon B, the second target is changed from B to C. Otherwise, the second target remains B. Even if Sebastian lost sight of balloon B, the location information of B is preserved, and after breaking A, it rotates in the direction of B. The direction of rotation is determined by the x and y coordinates of balloons A and B, and the relationship between yA/xA and yB/xB. Similarly, if a new balloon D is seen after balloon A is broken, the distance to balloon D is compared with the distance to balloon B, and the closer balloon B is selected as the next target. When breaking a balloon, it rushes toward the center coordinates of the balloon. Sebastian approaches the balloon, and when the balloon is too close to be recognized, Sebastian accelerates to gain momentum and break the balloon with the forward pin. The above is the basic flow of breaking a balloon, but in order to more accurately aim for a high score, the following algorithm was implemented. Based on the RGB value of the balloon’s center coordinates and the altitude at which the balloon exists, the algorithm recognizes what color the balloon is, and prioritizes high-scoring red balloons, while blue balloons with negative points are excluded from the target. Similarly, if the altitude at which a balloon exists exceeds the altitude of the water surface, it recognizes that the balloon is in a situation of total reflection on the water surface and does not aim this as a target. Similarly, the weight holding the balloon at the bottom of the pool is also excluded from the target because it is not at the altitude where the balloon should be, thus preventing Sebastian from mistaking the weight for a balloon.
AUV
This mission used the original cruising autonomous underwater vehicle “Sebastian” [Fig. 4]. This AUV is based on the wAriel AUV that participated in last year’s competition.
Sebastian is equipped with two Heave and two Surge thrusters and can be controlled in four degrees of freedom (Surge, Heave, Pitch and Yaw). The acrylic hull, which is a buoyant body, is positioned on the left and right sides and the center of gravity is designed in the center downwards to maintain stability in the roll direction. As sensors, a depth sensor for depth estimation and a camera module for recognizing balloons in the water were used. The AUV was also equipped with leg parts for landing on the bottom of the pool and guides for pushing the balloon against the pushpins from the front to destroy it.
Sebastian is controlled using a Raspberry Pi 4 computer and a Teensy driver. As a control tool, Sebastian uses a common open source tool for robotics control called Robot Operation System (ROS) on the Raspberry pi, and sends PWM commands to the motor drivers with the Teensy driver.
In this mission, circle detection is performed by OpenCV Hough transform of the USB camera image [Fig. 5]. The Hough gradient method can be used to recognize objects by changing the parameters and the processing of the image data to be captured. The results of balloon recognition using the blue part of the BGR image were output to an external PC for confirmation and showed sufficient recognition results to control the system. This shows that circle recognition using the Hough transform is effective for recognizing balloons in water.
The position of the balloon in the camera coordinate system was calculated from the size and position of the balloon as seen from the camera [2]. Assuming a balloon diameter of 20 cm, the relative position was calculated using the following equations (1), (2) and (3) [Fig. 6]. The actual depth and horizontal distance of the balloon was calculated by calculating the following equations (4), (5) and (6) using the internal IMU sensor and depth sensor.
For color recognition of balloons, the BGR score of the center of the balloon was used due to the computational limitations of the Raspberry Pi. If the center of the balloon was outside the camera’s field of view, the point in the field of view closest to the center was used for color recognition [Fig. 7]. The depth of the balloon was also used to aid color estimation.
The competition rules were presented in June 2023 and a development project was initiated. Software development was divided into balloon recognition and target determination algorithms, vehicle control and state transition algorithms, and firmware development. by July, hardware improvements and control code for minimum functions were completed. As of early August, balloons could be destroyed under a certain condition.
In August, debugging, parameter adjustments and exception processing were added to improve the robustness of the system. In parallel, the algorithm for determining which balloon to break next to the currently approached balloon was also implemented.
Finally, the color recognition success rate and the approach success rate were confirmed to be more than 90% against blue, red, yellow, green and pink balloons.
Result
On the first day of the convention, each team presented a poster session and answered questions from the judges and other participants. We also tested the operation of Sebastian in the water tank. The second day of the convention consisted of a water tank competition, which concluded with an awards ceremony. We were awarded first place out of all four teams for breaking the most balloons and for our high degree of autonomy to operate without tether cables.
Through this competition, we were able to realize the difficulty of underwater robot development, the importance of teamwork, and networking with people in the community. The experience we gained through our three months of intensive development and participation in the competition will certainly be a great source of inspiration for our future research activities.
Acknowledgement
The Underwater Robot Convention in JAMSTEC in 2023 was supported by The Japan Society of Naval Architects and Ocean Engineers, IEEE/OES Japan Chapter, MTS Japan Section, Techno-Ocean Network, Kanagawa Prefecture, Yokosuka City, Japan Agency for Marine-Earth Science and Technology (JAMSTEC), Center for Integrated Underwater Observation Technology at Institute of Industrial Science, the University of Tokyo, FullDepth Co., Ltd., Nortek Japan LLC, Japan Branch of Robotiz, Inc, ARAV Co., Ltd., Sea challenge Co., Ltd., Space Entertainment Laboratory Co., Ltd., Matsuyama Industry Co., Ltd., IWAKITEC Co., Ltd., Chick Co., Ltd., Misago Co., Ltd., and Aqua Modelers Meeting. We would like to express our sincere appreciation to the sponsors for their strong support and cooperation in realizing this convention.
Comments
Hiroshi Kasuga: I am pleased that we won, but there are many points that can be improved upon. I hope to make use of this experience in my research.
Lisa Hakataya: I’m glad we won. However, there were many things we could not do. We would like to increase what we can do in the future.
Hiroki Yokohata: I’m very happy to have achieved another great result of winning the first place this year, as we did last year. I would like to thank my teammates for leading the team to this height.
References
[1] Underwater Robot Convention in JAMSTEC 2023 (Japanese). http://jam23.underwaterrobonet.org/ [2] A. Toriyama, M. Ohashi, H. Yokohata, wARIEL, the AUV Won First Place in Underwater Robot Convention in JAMSTEC 2022! IEEE OES Beacon Newsletter, 11(4), 83-86 (2022.12) [3] K. Yamamoto, S. Chun, Y. Sekimori, C. Kawamura, ARIEL, the AUV Won First Place in Underwater Robot Convention in JAMSTEC 2021! IEEE OES Beacon Newsletter, 10(4), 70-73 (2021.12) [4] Y. Sekimori, T. Maki, Underwater Robot Convention in JAMSTEC 2020 – All Hands on Deck! Online!!, IEEE OES Beacon Newsletter, 10(1), 39-42 (2021.3) [5] K. Fujita, Y. Hamamatsu, H. Yatagai, Reflection for Singapore Autonomous Underwater Vehicle Challenge – the Comparison Between SAUVC and a Competition Held in Japan, IEEE OES Beacon Newsletter, 8(2), 64-67 (2019.6) [6] H. Yamagata, T. Maki, Underwater Robot Convention in JAMSTEC 2018 – from an Educational Perspective, IEEE OES Beacon Newsletter, 7(4), 68-72 (2018.12)
Suleman Mazhar has been working as a professor in Information & Communication Engineering at Harbin Engineering University (China) since July 2019. He did PhD from Tokyo University (Japan) and postdoctorate from Georgetown University (Washington DC, USA). He had BS-CS from FAST-NUCES (Lahore) and MS from GIK Institute (Pakistan). He is TYSP young scientist fellow (Ministry of Science & Technology China) and have won several research grants from international organizations such as DAAD (Germany), ICIMOD (Nepal), NRPU (Higher Education Commission Pakistan), WWF (Worldwide Fund for Nature) Pakistan. His research focus is deep learning and signal processing applications for environmental monitoring, with particular focus on underwater acoustics, and marine mammal conservation. He is a reviewer for professional journals such as Journal of Acoustical Society (America), IEEE Journal of Oceanic Engineering, IEEE Sensors Journal, Applied Acoustics, IEEE Transactions on Intelligent Transportation Systems.
Peng Ren is a full professor with the College of Oceanography and Space Informatics, China University of Petroleum (East China). He is the director of Qingdao International Research Center for Intelligent Forecast and Detection of Oceanic Catastrophes. He received the K. M. Scott Prize from the University of York, the Natural Science award (first rank) from China Institute of Electronics, and the Eduardo Caianiello Best Student Paper Award from 18th International Conference on Image Analysis and Processing as one co-author. He has served as an associate editor of IEEE Transactions on Geoscience and Remote Sensing.
Mohd Rizal Arshad is a full professor at the School of Electrical and Electronic Engineering at Universiti Sains Malaysia (USM), Malaysia, where he specializes in ocean robotics technology and intelligent system. He received his B.Eng. in Medical Electronics & Instrumentation and PhD in Electronic Engineering from University of Liverpool, UK in 1994 and 1999, respectively. He completed his MSc. in Electronic Control Engineering from the University of Salford, UK in Dec 1995. He has supervised many postgraduate students and published extensively in local and international publications. He is a senior member of the IEEE, and was awarded IEEE OES Presidential Award in 2019.
Itzik Klein is an Assistant Professor, heading the Autonomous Navigation and Sensor Fusion Lab, at the Charney School of Marine Sciences, Hatter Department of Marine Technologies, University of Haifa. He is an IEEE Senior Member and a member of the IEEE Journal of Indoor and Seamless Positioning and Navigation (J-ISPIN) Editorial Board. Prior to joining the University of Haifa, he worked at leading companies in Israel on navigation topics for more than 15 years. He has a wide range of experience in navigation systems and sensor fusion from both industry and academic perspectives. His research interests lie in the intersection of artificial intelligence with inertial sensing, sensor fusion, and autonomous underwater vehicles.
John R. Potter (IEEE M’94, SM’02, F’18) graduated in the previous century with a joint honours Mathematics and Physics Degree from Bristol and a PhD. in Glaciology and Oceanography from Cambridge, UK studying Antarctic ice mass balance, where he spent four consecutive summers. This work helped underscore the non-linear fragility of polar ice to climate change and led to him receiving the Polar Medal from Queen Elizabeth II in 1988.
Nick is a Visiting Fellow at the UK National Oceanographic Center, Southampton His nomination was endorsed by the Underwater Acoustics Technology Committee. He had worked as a Research Associate and Lecturer at University of Birmingham and has been working as a Research Scientist at the Applied Research Laboratory, University of Texas, Austin. He has also served as a Program Officer at the Office of Naval Research Global. He is a senior member of IEEE (OES) and a Fellow of Acoustical Society of America (ASA). Nick has also been serving as Assoc. Editor for IEEE JoE and JASA. He is widely acknowledged for his expertise are seabed acoustics, parametric array modeling, sonar beamformer, underwater signal processing.
Maurizio Migliaccio (M’91-SM’00-F’17) is Full professor of Electromagnetics at Università di Napoli Parthenope (Italy) and was Affiliated Full Professor at NOVA Southeastern University, Fort Lauderdale, FL (USA). He has been teaching Microwave Remote Sensing since 1994. He was visiting scientist at Deutsche Forschungsanstalt fur Lüft und Raumfahrt (DLR), Oberpfaffenhofen, Germany. He was member of the Italian Space Agency (ASI) scientific committee. He was member of the ASI CosmoSkyMed second generation panel. He was e-geos AdCom member. He was Italian delegate of the ESA PB-EO board. He was Member of South Africa Expert Review Panel for Space Exploration. He serves as reviewer for the UE, Italian Research Ministry (MIUR), NCST, Kazakhstan and Hong Kong Research board. He lectured in USA, Canada, Brazil, China, Hong Kong, Germany, Spain, Czech Republic, Switzerland and Italy. He was Italian delegate at UE COST SMOS Mode Action. He is listed in the Italian Top Scientists. He is an IEEE Trans. Geoscience and Remote Sensing AE, International Journal of Remote Sensing AE, and was IEEE Journal of Oceanic Engineering AE Special Issue on Radar for Marine and Maritime Remote Sensing, IEEE JSTARS AE of the Special Issue on CosmoSKyMed, Member of the Indian Journal of Radio & Space Physics Editorial board. His main current scientific interests cover SAR sea oil slick and man-made target monitoring, remote sensing for marine and coastal applications, remote sensing for agriculture monitoring, polarimetry, inverse problems for resolution enhancement, reverberating chambers. He published about 160 peer-reviewed journal papers on remote sensing and applied electromagnetics.
He has developed various types of Autonomous Underwater Vehicles (AUVs) and related application technologies including navigation methods, a new sensing method using a chemical sensor, precise seafloor mapping methods, a precise seabed positioning system with a resolution of a few centimeters, a new sensing system of the thickness of cobalt-rich crust; and more. He has shown, by using these technologies that AUVs are practicable and valuable tools for deep-sea exploration.
Donna Kocak has had an outstanding career in defense and scientific projects developing and applying solutions in subsea optics, imaging and robotics. She graduated with an M.Sc in Computer Science in 1997 from the University of Central Florida; an MBA in 2008 from the University of Florida; and M.Sc in Industrial Engineering in 2011 from the University of Central Florida. She is currently a Senior Scientist, Advanced Concepts Engineering, and Fellow at the Harris Corporation in Melbourne, Florida, where she has developed novel optical imaging and communication solutions for under-sea defense and scientific projects. Prior to 2008 Donna Kocak was Founder and President of Green Sky Imaging, LLC (GSI) who developed laser/video photogrammetry software for underwater inspection and survey. Her earlier career positions were with Naval Training Systems Center, Florida; Harbor Branch Oceanographic Institution, Florida; eMerge Interactive; and the Advanced Technologies Group in Florida.
John Potter has a Joint Honours degree in Mathematics and Physics from Bristol University in the UK and a PhD in Glaciology and Oceanography from the University of Cambridge on research in the Antarctic, for which he was awarded the Polar Medal in 1988. John has worked on polar oceanography, underwater acoustics, ambient noise (including imaging), marine mammals, communications, IoUT, autonomous vehicles and strategic development. He has 40 years’ international experience working at the British Antarctic Survey in the UK, NATO in Italy, SIO in California, NUS in Singapore and most recently at NTNU in Norway. John is a Fellow of the IEEE and MTS, an Associate Editor for the IEEE Journal of Oceanic Engineering, IEEE OES Distinguished Lecturer, PADI Master Scuba Diver Trainer & an International Fellow of the Explorer’s Club.
Dr. James V. Candy is the Chief Scientist for Engineering and former Director of the Center for Advanced Signal & Image Sciences at the University of California, Lawrence Livermore National Laboratory. Dr. Candy received a commission in the USAF in 1967 and was a Systems Engineer/Test Director from 1967 to 1971. He has been a Researcher at the Lawrence Livermore National Laboratory since 1976 holding various positions including that of Project Engineer for Signal Processing and Thrust Area Leader for Signal and Control Engineering. Educationally, he received his B.S.E.E. degree from the University of Cincinnati and his M.S.E. and Ph.D. degrees in Electrical Engineering from the University of Florida, Gainesville. He is a registered Control System Engineer in the state of California. He has been an Adjunct Professor at San Francisco State University, University of Santa Clara, and UC Berkeley, Extension teaching graduate courses in signal and image processing. He is an Adjunct Full-Professor at the University of California, Santa Barbara. Dr. Candy is a Fellow of the IEEE and a Fellow of the Acoustical Society of America (ASA) and elected as a Life Member (Fellow) at the University of Cambridge (Clare Hall College). He is a member of Eta Kappa Nu and Phi Kappa Phi honorary societies. He was elected as a Distinguished Alumnus by the University of Cincinnati. Dr. Candy received the IEEE Distinguished Technical Achievement Award for the “development of model-based signal processing in ocean acoustics.” Dr. Candy was selected as a IEEE Distinguished Lecturer for oceanic signal processing as well as presenting an IEEE tutorial on advanced signal processing available through their video website courses. He was nominated for the prestigious Edward Teller Fellowship at Lawrence Livermore National Laboratory. Dr. Candy was awarded the Interdisciplinary Helmholtz-Rayleigh Silver Medal in Signal Processing/Underwater Acoustics by the Acoustical Society of America for his technical contributions. He has published over 225 journal articles, book chapters, and technical reports as well as written three texts in signal processing, “Signal Processing: the Model-Based Approach,” (McGraw-Hill, 1986), “Signal Processing: the Modern Approach,” (McGraw-Hill, 1988), “Model-Based Signal Processing,” (Wiley/IEEE Press, 2006) and “Bayesian Signal Processing: Classical, Modern and Particle Filtering” (Wiley/IEEE Press, 2009). He was the General Chairman of the inaugural 2006 IEEE Nonlinear Statistical Signal Processing Workshop held at the Corpus Christi College, University of Cambridge. He has presented a variety of short courses and tutorials sponsored by the IEEE and ASA in Applied Signal Processing, Spectral Estimation, Advanced Digital Signal Processing, Applied Model-Based Signal Processing, Applied Acoustical Signal Processing, Model-Based Ocean Acoustic Signal Processing and Bayesian Signal Processing for IEEE Oceanic Engineering Society/ASA. He has also presented short courses in Applied Model-Based Signal Processing for the SPIE Optical Society. He is currently the IEEE Chair of the Technical Committee on “Sonar Signal and Image Processing” and was the Chair of the ASA Technical Committee on “Signal Processing in Acoustics” as well as being an Associate Editor for Signal Processing of ASA (on-line JASAXL). He was recently nominated for the Vice Presidency of the ASA and elected as a member of the Administrative Committee of IEEE OES. His research interests include Bayesian estimation, identification, spatial estimation, signal and image processing, array signal processing, nonlinear signal processing, tomography, sonar/radar processing and biomedical applications.
Kenneth Foote is a Senior Scientist at the Woods Hole Oceanographic Institution. He received a B.S. in Electrical Engineering from The George Washington University in 1968, and a Ph.D. in Physics from Brown University in 1973. He was an engineer at Raytheon Company, 1968-1974; postdoctoral scholar at Loughborough University of Technology, 1974-1975; research fellow and substitute lecturer at the University of Bergen, 1975-1981. He began working at the Institute of Marine Research, Bergen, in 1979; joined the Woods Hole Oceanographic Institution in 1999. His general area of expertise is in underwater sound scattering, with applications to the quantification of fish, other aquatic organisms, and physical scatterers in the water column and on the seafloor. In developing and transitioning acoustic methods and instruments to operations at sea, he has worked from 77°N to 55°S.
René Garello, professor at Télécom Bretagne, Fellow IEEE, co-leader of the TOMS (Traitements, Observations et Méthodes Statistiques) research team, in Pôle CID of the UMR CNRS 3192 Lab-STICC.
Professor Mal Heron is Adjunct Professor in the Marine Geophysical Laboratory at James Cook University in Townsville, Australia, and is CEO of Portmap Remote Ocean Sensing Pty Ltd. His PhD work in Auckland, New Zealand, was on radio-wave probing of the ionosphere, and that is reflected in his early ionospheric papers. He changed research fields to the scattering of HF radio waves from the ocean surface during the 1980s. Through the 1990s his research has broadened into oceanographic phenomena which can be studied by remote sensing, including HF radar and salinity mapping from airborne microwave radiometers . Throughout, there have been one-off papers where he has been involved in solving a problem in a cognate area like medical physics, and paleobiogeography. Occasionally, he has diverted into side-tracks like a burst of papers on the effect of bushfires on radio communications. His present project of the Australian Coastal Ocean Radar Network (ACORN) is about the development of new processing methods and applications of HF radar data to address oceanography problems. He is currently promoting the use of high resolution VHF ocean radars, based on the PortMap high resolution radar.
Hanu Singh graduated B.S. ECE and Computer Science (1989) from George Mason University and Ph.D. (1995) from MIT/Woods Hole.He led the development and commercialization of the Seabed AUV, nine of which are in operation at other universities and government laboratories around the world. He was technical lead for development and operations for Polar AUVs (Jaguar and Puma) and towed vehicles(Camper and Seasled), and the development and commercialization of the Jetyak ASVs, 18 of which are currently in use. He was involved in the development of UAS for polar and oceanographic applications, and high resolution multi-sensor acoustic and optical mapping with underwater vehicles on over 55 oceanographic cruises in support of physical oceanography, marine archaeology, biology, fisheries, coral reef studies, geology and geophysics and sea-ice studies. He is an accomplished Research Student advisor and has made strong collaborations across the US (including at MIT, SIO, Stanford, Columbia LDEO) and internationally including in the UK, Australia, Canada, Korea, Taiwan, China, Japan, India, Sweden and Norway. Hanu Singh is currently Chair of the IEEE Ocean Engineering Technology Committee on Autonomous Marine Systems with responsibilities that include organizing the biennial IEEE AUV Conference, 2008 onwards. Associate Editor, IEEE Journal of Oceanic Engineering, 2007-2011. Associate editor, Journal of Field Robotics 2012 onwards.
Milica Stojanovic graduated from the University of Belgrade, Serbia, in 1988, and received the M.S. and Ph.D. degrees in electrical engineering from Northeastern University in Boston, in 1991 and 1993. She was a Principal Scientist at the Massachusetts Institute of Technology, and in 2008 joined Northeastern University, where she is currently a Professor of electrical and computer engineering. She is also a Guest Investigator at the Woods Hole Oceanographic Institution. Milica’s research interests include digital communications theory, statistical signal processing and wireless networks, and their applications to underwater acoustic systems. She has made pioneering contributions to underwater acoustic communications, and her work has been widely cited. She is a Fellow of the IEEE, and serves as an Associate Editor for its Journal of Oceanic Engineering (and in the past for Transactions on Signal Processing and Transactions on Vehicular Technology). She also serves on the Advisory Board of the IEEE Communication Letters, and chairs the IEEE Ocean Engineering Society’s Technical Committee for Underwater Communication, Navigation and Positioning. Milica is the recipient of the 2015 IEEE/OES Distinguished Technical Achievement Award.
Dr. Paul C. Hines was born and raised in Glace Bay, Cape Breton. From 1977-1981 he attended Dalhousie University, Halifax, Nova Scotia, graduating with a B.Sc. (Hon) in Engineering-Physics.