COSMO

The COSMO project aimed to research and develop data-driven techniques that would enable more efficient content creation for assembly training for AR and VR applications and allow such applications to adapt to the skill level of the individual operator. The project also investigated the impact of AR and VR training on operators’ learning processes and on stress levels.

The Outcomes

1. (Partially) automated content creation

COSMO’s vision to tackle the cost of content creation was that audiovisual recordings of an expert demonstrating the assembly process step-by-step would be translated into digital work instructions that can be ingested by VR or AR training applications. The project has developed a proof-of-concept solution that partially automates this process, reducing the time needed for producing training content by 50% on average. Through a clever combination of image recognition and language technologies, raw recordings of the expert’s demonstration are analyzed and segmented into sets of digital work instructions. This eliminates the need for manually placing all the parts and tools in a VR scene, and for manually defining the assembly sequence in the training tool. Components of products and workstations can be added through CAD models to improve the image recognition. 

2. Data-driven personalization of training

Overabundant instructions in trainings can reduce the opportunities for longer-term learning and even reduce productivity. On the other hand, too little detail in the work instructions is equally ineffective and can induce stress. COSMO’s solution estimates the skill levels of the individual operator from moment to moment and delivers tailored instructions that gradually fade out as the operator becomes more skilled with the task at hand. For example, a new operator may receive full audiovisual explanations initially transitioning to gentle corrective nudges as they become able to perform an assembly task more independently. COSMO also developed an efficient machine learning model for estimating the stress levels of operators based on state-of-the-art wearable sensors and transfer learning techniques.

3. Integrated AR and VR applications for evaluation

These techniques were integrated into applications for training operators how to assemble a variety of products such as the central beam of agricultural vehicles (VR) or a wooden birdhouse (AR). The learning effectiveness of these applications was evaluated in realistic environments, with operators in manufacturing companies and students in upper-secondary technical, vocational and special education. Results from scientific experiments show that data-driven adaptive work instructions reduce assembly times and mental effort. More generally, the experiments show that when the work instructions are soundly designed based on cognitive models for training and evidence in the learning sciences, immersive assembly training with AR and VR has large effects on assembly times, perceived complexity and mental effort compared with paper-based training.