Science and Tech
By: Lina María Leal Villamizar
Photos: Óscar Perdomo. DOI https://doi.org/10.12804/dvcn_10336.37600_num6
According to Instituto Nacional para Sordos (National Institute for the Deaf - Insor is the Spanish acronym), in 2020, 554,119 people with hearing disabilities were registered in Colombia, out of which only 42.6% school-age children (ages 5 to 16) were enrolled in the educational system. In other words, most Colombian deaf kids and adolescents do not receive formal education at an academic institution.
Robins is covered with plush because children usually feel more attracted toward soft textures they can touch and that are similar to their favorite toys.
Why does this happen? Speech pathologist Rocio Stella Molina Bejar, professor from the School of Medicine and Health Sciences at Universidad del Rosario with a Master of Sciences degree in Disabilities and Social Inclusion, claims that the reading and writing learning process is difficult for the deaf population since it mostly depends on whether hearing loss was discovered at its early stages. Even so, in most cases, children end up communicating only through sign language, either Colombia’s own or the sign language that they have created within their family or social bubble, without properly learning to read and write Spanish.
Molina Bejar explains that sign languages are not universal. Instead, they change from culture to culture since each culture may have their own particular expressions for constructing ideas, just as it happens with any spoken language. Likewise, many families with hearing parents and deaf children build their own internal communication systems.
However, when children first go to school and start learning to read and write, they discover, for example, that sign languages do not use pronouns or plurals, while written (and oral) communication does. For this reason, according to Molina, who has been working with this population for over 20 years, literacy generates “conflict” for these students.
Within this context, Karen Aguia Rojas, a teacher at the Occupational Therapy program, warns that the problem is bidirectional: “People without hearing disabilities lack the elements to be able to talk and communicate with that specific population. We are asking deaf kids to learn Spanish, but we are not learning sign language. Then, what are we doing to communicate with them?”
A group of researchers, including Molina and Aguía, have assessed this situation with a magnifying glass. This process was pioneered by Oscar Julián Perdomo Charry, a professor from the Biomedical Engineering program at Universidad del Rosario, who considered building an Artificial Intelligence (AI) device for educating vulnerable populations.
“We observed different virtual assistants on the market, but they were all voice activated, such as Siri or Alexa, among others. We walked a mile in the shoes of someone who cannot speak and we realized that these assistants are not functional for them,” explains Perdomo, who along with Hernan Bernal and Alvaro David Orjuela, members of the EduBraille group, decided in 2021 to form a team aimed at finding possible technological solutions for the hearing-impaired population.
As engineers experimented in the labs with devices based on cameras and electronic cards, other researchers from different areas, such as Health Sciences, Occupational Therapy (Karen Aguía Rojas), Psychology (Sara CaDavid Espinha; Oliver Muller), and Speech Therapy (Rocío Molina Bejar) also started to heed the call. Even though these disciplines seemed very different from other, they all shared the same common ground: “Technology changes people’s lives. That was the rationale that brought us all together; being able to see how these innovations can help people with disabilities is wonderful,” says speech therapist Molina.
Hence, everyone contributed to the success of the project. Engineering, led by Professor Perdomo and Hernan Bernal, a Biomedical Engineering laboratory assistant, continued unravelling the programming and mechanical aspects to continuously improve the articulation of the prototype.
Meanwhile, the Health Sciences team studied the requirements for this device and wondered, What can be the best educational strategy to challenge? What does a child require and what can they do? What are the environmental elements needed to improve or maintain the child’s motivation and what are the child’s own elements? What are the instruments available to assess the child’s learning process or motivation?
Based on this, throughout 2021 and 2022, this team conducted a rigorous study aimed at determining what has been done in the world in this regard. As part of their results, they discovered that although there are supporting devices for the deaf community, they need to be imported and their price is very high. In Colombia, there is little social innovation arising from Engineering and Health Sciences in the development of proprietary technology for deaf people.
Furthermore, the team collaborated with two Colombian institutions that treat children with hearing loss: the ICAL foundation for deaf children (ICAL) and Colegio Filadelfia, a Colombian school for the deaf. This helped them to better understand their learning needs and challenges faced by Spanish language teachers. In addition, they were able to learn more about existing pedagogical strategies for working with this particular population.
Professor Óscar Julián Perdomo Charry, from the Biomedical Engineering program at Universidad del Rosario, considered building an artificial intelligence device aimed at educating vulnerable populations: Robins.
Together they created a functional assistant named Robins, an interactive robot for deaf children. This technology supplements the reading learning process with an appropriate strategy specifically designed for children with hearing disabilities. For instance, the team programmed automatic facial expressions and sign language recognition into the device to ensure proper interactions.
According to Perdomo, Robins’ structure was entirely printed using a 3D printer at the Universidad del Rosario, which means that it is affordable, portable, and customizable to individual needs. “The hardware comprises printed parts that can be adjusted depending on the needs of each particular user,” the researchers explain.
The initial prototype is a vertebrate animal that protects and holds a depth camera, a touch screen, and an electronic card that automatically recognizes facial expressions and the sign language used by deaf children based on artificial intelligence algorithms.
Its upper structure (head) holds three motors for its eyebrows, cheekbones, and mouth. In addition, the robot’s eyes have two color LED screens and an additional screen used to customize its eyes and their movement. Aguía comments that “adding eyes and eyebrows to the robot was critical because they are the main components used for transmitting emotions.”
In addition, Robins is covered with plush because according to the Occupational Therapy and Psychology researchers, children usually feel more attracted toward soft textures they can touch and that are similar to their favorite toys. The team is still thinking about whether Robins should be shaped into a dog, a teddy bear, a cat, or a koala bear.
In its belly, Robins also has a camera that detects what the child is doing and a screen that projects images. The idea is that the robot can interact based on what it captures. “We have recorded videos of people signing the alphabet so that the ‘dog’ may learn to recognize certain signs as letters of the fingerspelling alphabet, the manual alphabet used by deaf-blind people,” comments neuropsychologist Cadavid Espinha, who adds that the robot may develop different phases at increasing difficulty (several sessions each) throughout the reading and writing teaching process.
Initially, the robot displays an image that corresponds perfectly with the text that is included, and the child must mark one of the letters of the word. For example, the robot displays the image of a bee along with the corresponding Spanish word (“abeja”) underneath. The first letter of the word will be displayed in red.
The student will have to fingerspell the letter A. Thereafter, the child will be prompted to sign the remaining letters of the word.
Later, as the following phase, the robot will display signs recorded by a deaf person, so the student can work with more abstract words, such as “freedom,” “justice,” etc. Again, the corresponding word in Spanish will be displayed underneath, and the student will be prompted to fingerspell the letter displayed in red.
In the future, the research team hopes that the technology consolidated in Robins will be able to develop other tasks, such as helping children exercise and even interact in groups, always hand in hand with education, its most innovative function.
Finally, the level of difficulty increases as the number of images and signs also increase, and the children start identifying more than one letter in a word. “The challenge is for each deaf boy or girl to start associating what they see (the images) with fingerspelling each letter or word,” comments Cadavid.
The child will sign after each exercise, and the robot will immediately provide feedback with gestures or images: “Good!” or “Are you sure?” or a smiley face for correct answers and a surprised face whenever the answer is wrong. The robot will also display fun-filled animations and use a point-based system to encourage the child to obtain a prize or reward, such as playing with his/her favorite toy.
However, considering that all of this could also be done by a human, what advantage does the robot offer? Cadavid responds that “technology constitutes that extra mile or added value to motivate children since the learning process usually generates some resistance and with this technology, it can become more stimulating.”
Professor Oliver Müller, from the Psychology program and who has previously worked on reading projects with deaf children, is also part of the team and points out that the robot seeks to foster motivation toward learning in a more playful way: “Not as your typical elementary education teacher but as an additional element, one that provides feedback with facial gestures and games.”
The researchers plan to have Robins measure children’s motivation through data collection instruments applied before and after using the technology. “We expect the robot to act as a mediator that can foster the child’s motivation toward learning,” says Professor Aguía.
Initially, the robot displays an image that corresponds perfectly to the text that is included, and the child must mark one of the letters of the word.
For example, one of the tools measures 14 possible behaviors, among which are the child’s imagination about new things, his/her proposed strategies, or his/ her level of curiosity, among others. “Furthermore, we are working on how we finetune the activities in terms of difficulty: if an activity is very simple, it will not motivate the child, but if it is extremely difficult, it will generate frustration,” warns Aguía. “There has to be a match between the child’s abilities and the characteristics of assistive technology.”
The researchers acknowledge that while the robot is almost ready for pilot testing in the real world, it has not yet been tested with children. These tests will be conducted during the first academic semester of 2023. This will also allow researchers to conduct a first statistical analysis with data. Müller indicates that the instruments will determine whether the robot is fulfilling the expected motivator role. “Otherwise, Robins’ appeal (its ability to generate attraction) would have to be increased,” he concludes.
Although this could happen, speech pathologist Molina is confident that there will be positive changes. For example, she believes that considering the robot’s potential as an innovative and easily accessible technology, its daily use will surely improve the quality of life of deaf people. In fact, the initial phase of the project obtained recognition by becoming the only Colombian finalist in the University Team modality at the 2021 OpenCV AI Competition, a global competition sponsored by Microsoft Azure, Intel, and OpenCV (Open Source Computer Vision and Machine Learning Software Library).
This is the world’s largest artificial intelligence competition, wherein researchers demonstrated the capabilities of both Colombia and Universidad del Rosario in this area of knowledge.
The research team wants the construction plans to be freely accessible so that anyone can print the device in a 3D printer and assemble it. According to Müller and Perdomo, there are some very good social robots on the world market, but they are also very expensive because they are purely commercial products. They conclude as follows: “This project has taught us to become more empathetic.”
In the future, the research team hopes that the technology consolidated in Robins will be able to develop additional tasks, such as helping children exercise and even interact in groups, always hand in hand with education, its most innovative function. Moreover, Cadavid emphasizes that “even when there are plenty of social robots, very few contribute to the development of reading skills in deaf children. In this sense, we could say that Robins is quite special, perhaps even ‘unique’ since it motivates children with this disability to approach the universe of the letters of the alphabet in a more playful way.”
