New augmented reality tech can turn any surface into keyboard - Hi Tech & Innovation - Machine learning & AI

Computer science doctoral student Daniel Honrales demonstrates PropType, a patent-pending technology that overlays an augmented keyboard surface onto handheld objects. Credit: University of Texas at Dallas

Virtual keyboards are a frequent source of frustration for augmented reality (AR) users. The virtual surfaces are slow and error prone, and raising an arm to type on them can cause muscle strain known as "gorilla arm."

To improve virtual-typing experiences, University of Texas at Dallas researchers have designed a unique interface that allows users to transform everyday objects into typing surfaces within an AR environment.

The patent-pending technology, called PropType, overlays an augmented keyboard surface onto a handheld object and can adapt to curved surfaces.

Student researchers have created a video demonstrating PropType being used on surfaces such as water bottles, coffee cups, books and soda cans. 

"By integrating objects already present in the user's surroundings, PropType reimagines text input in AR, creating a seamless connection between the physical and virtual worlds," said Dr. Jin Ryong Kim, assistant professor of computer science in the Erik Jonsson School of Engineering and Computer Science. "PropType capitalizes on the tactile feedback provided by the objects themselves, offering better key confirmation and reducing reliance on visual cues."

The technology provides an alternative to common AR typing solutions such as external physical keyboards, which can disrupt a device-free immersive experience.

"PropType leverages the familiarity of handheld objects to offer a more intuitive and accessible alternative to traditional keyboards, particularly in mobile or hands-free scenarios where conventional input methods are impractical," Kim said.

Researchers in Kim's Multimodal Interaction Lab presented PropType and received a Best Paper Honorable Mention Award in April at the ACM CHI Conference on Human Factors in Computing Systems, the world's premier conference in human-computer interaction, held in Yokohama, Japan. They later showcased the work in September at the 38th Annual Association for Computing Machinery (ACM) Symposium on User Interface Software and Technology in Busan, South Korea.

Daniel Honrales uses tracking marker rigs, the objects on his fingers and attached to the mug, to capture movement on a virtual keyboard. Credit: University of Texas at Dallas

Developing PropType involved challenges such as adapting the interface to different shapes and sizes of objects. The researchers studied how 16 participants interacted with props to understand grab postures and typing gestures. Then they developed custom keyboard layouts for different objects. The researchers included an editing tool that allows users to customize keyboard layouts and visual effects.

Kim's research focuses on haptics, or touch and physical feedback in digital systems, which can include vibrations and heat.

"Touch carries a lot of information; it's another form of communication that is underexplored in virtual and augmented reality," he said.

Kim and student researchers have received honors for other projects, including the Best Demo Award at the IEEE World Haptics Conference 2025 and a Best Demo Honorable Mention at the IEEE Haptics Symposium 2024 for their work on thermal masking and thermal-tactile integration.

Thermal masking is a phenomenon that tricks the brain into feeling heat or cold in a specific spot on the body, even though the source of the heat or cold is generated from a different location. For example, researchers place a heat actuator—a device that produces heat—on a user's arm and a tactile actuator, which produces vibrations, on the same arm several inches from the heat actuator. When researchers activate both actuators, the brain senses the heat at both locations.

The researchers presented a paper on the subject at the 2024 ACM CHI Conference on Human Factors in Computing Systems and created a video to demonstrate the technology.

"By delving into how tactile sensations can override thermal perceptions, we aim to create more immersive and responsive user interfaces," Kim said. "Our work holds promise for revolutionizing applications in virtual reality, medical simulations and beyond to create experiences that feel almost tangible."

Provided by University of Texas at Dallas 

by Kim Horner, University of Texas at Dallas

edited by Gaby Clark, reviewed by Robert Egan 

Source: New augmented reality tech can turn any surface into keyboard