Realistic visuals and conversational AI deliver natural communication

Tokyo – Toppan, a global leader in communication, security, packaging, décor materials, and electronics solutions, and 3dig inc. (3dig) are collaborating on using high-resolution real scanning data in the field of high-definition virtual humans. The two companies have developed “VH Signage,” a system allowing users to converse and interact with life-size, photorealistic virtual humans created using automated generation technology. Toppan and 3dig will launch a pilot test in April this year at Toppan’s Kanda Izumi Building in Chiyoda, Tokyo. The test will look into how people are affected by interacting with virtual humans designed to be more realistic, their feelings about it, and how to ensure the best possible user experience.

The VH Signage system consists of servers, monitors, and sensors. In addition to photorealistic visuals, it includes conversational artificial intelligence (AI) using synthesized speech and has interaction functions. By interacting with the VH Signage as if they were talking to a real person, users will be able to enjoy a range of services, including guidance and product purchases.

“By creating photorealistic virtual humans and offering interactive signage, we want to deliver something that is user-friendly for everyone,” said Shu Tanabe, Business Innovation Manager in Toppan’s Business Development Division. “This pilot test should demonstrate the effectiveness of the system while also enabling us to identify what we need to do to provide the best possible solution going forward.”

Background

Recent years have seen robots and signage systems handling reception and guidance functions in stations and commercial facilities to prevent the spread of infectious diseases and reduce the human workload. However, issues that have emerged include senior citizens being reluctant to use touch panel signage and the breakdown of users being slanted towards certain demographics in the case of AI audio guidance using animated characters or automated speech. This has created a need for more inclusive solutions, catering to every kind of user, especially in public spaces.

In response, Toppan and 3dig have developed the VH Signage system, which combines life-size, photorealistic virtual humans with conversational AI using synthetic speech. High-definition data from biometric scanning on the Light Stage1 at the Toppan Virtual Human Lab2 is fused with 3dig’s AI-powered “anma” system for automated generation to enable efficient production of virtual human versions of real people (“real VH”). A high-definition, photorealistic “real VH” can be combined with conversational AI to deliver a natural experience similar to conversing with the actual person.

Toppan and 3dig intend to provide VH Signage and other solutions using high-definition virtual humans to commercial facilities, local governments, and a range of other customers and to develop a business focused on managing data obtained from the scanning of people on whom virtual humans are modeled.

Features of VH Signage

?Natural communication facilitated by life-size, photorealistic virtual humans that can start conversations and follow the user’s line of sight
Life-size, photorealistic virtual humans appear on 86-inch large format signage (D35.7 x W192.5 x H117 cm). Built-in human recognition sensors enable the virtual humans to communicate with users in front of the screen by facing and talking to them. This delivers a natural experience similar to being spoken to by a real person and makes it possible for virtual humans to provide users with diverse services through conversation that replicates everyday human interaction.

?Efficient production of high-definition virtual humans enabled by scanning and automation technologies
The Toppan Virtual Human Lab and its Light Stage are used to create photorealistic virtual humans based on data from high-definition scanning of the human body. Virtual humans based on actual people can be produced with shorter lead times and at lower cost by using 3dig’s anma system for automated generation.

Overview of pilot test

-Period: April 13 to 27, 2023
-Location: First floor reception area at Toppan’s Kanda Izumi Building (Chiyoda, Tokyo)
-Objective: To verify the effectiveness of interaction using VH Signage based on real people.
-Details: Reception and guidance functions will be handled by VH Signage in the reception area of Toppan’s building (assuming guidance provided to ten visitors per day). Specifically, life-size virtual humans modeled on real people will appear on large, 86-inch signage. The test will seek to verify the best way to use VH Signage to handle part of reception work and reduce human workload by greeting, talking to, and actively interacting with visitors.
-Target: The VH Signage will interact with visitors, Toppan new recruits considered to be members of Generation Z, and employees aged 60 and above. The pilot test will be explained to participants and their consent will be obtained prior to taking part.
-Method: Participants will be asked to respond to a simple questionnaire, based on which quantitative and qualitative analysis will be performed to guide future functional enhancement as well as consideration of quality and cost (pricing) for commercialization.

Roles of the companies

Toppan
Toppan will design and implement the pilot test, including high-definition scanning of real people’s faces using a Light Stage and conversion to three-dimensional computer graphics.

3dig
3dig will develop the VH Signage system and use its automated virtual human generation system, anma, to produce virtual humans modeled on people scanned by Toppan and load them into the VH Signage.
3dig website: https://3dig.net/ (in Japanese)
Website for anma, 3dig’s automated generation system: https://3dig.net/anma/ (in Japanese)

1. A Light Stage is a piece of equipment for high-precision face scanning developed by the University of Southern California.
2. The Toppan Virtual Human Lab was opened in December 2020 as a facility for advancing research and application development related to the use of face scanning and various other human biometric data.