Immersive Technology With Mixed Reality for Manufacturing Industry

How can Mixed Reality and Microsoft HoloLens help track malfunctions and compliance issues in Manufacturing?

By Pravin Dhayfule, Senior Manager, UX COE

Mixed Reality: The Shifting Focus in Manufacturing

The global pandemic impacted several industries including the worldwide manufacturing industry. The business problems that showed up have made the industry to rethink workflows by introducing digital technology solutions. Mixed Reality (MR) can be a solution to the pressing remote working challenges in the manufacturing sector.

As per the IDC Worldwide Augmented and Virtual Reality Spending Guide (Forecast 2020), commercial use cases that will see large investments in 2024 – $4.1 billion for industrial maintenance, $4.1 billion for training and $2.7 billion for retail showcasing.

In the immersive tech space, Mixed Reality expands the potential of Augmented Reality (AR) by allowing the user to not just see the virtual environment with a smart glass or headset but also interact with it using hand gestures. Leading brands like Mercedes-Benz, Airbus and many others are already utilizing Mixed Reality with wearables like Microsoft HoloLens to improve technician support, increase production and achieve various business benefits.

This blog covers the various business aspects of using Mixed Reality to overcome the hurdles and gaps caused due to the Global Pandemic in the Manufacturing Industry. We also explore a manufacturing use case in detail.

1. The Unforeseen Challenges in Global Manufacturing

As per a UN report published in 2020, there has been a 9% year-on-year decline in the global production and manufacturing output. The manufacturing industry was already facing issues due to trade tensions and the pandemic led to further decline.

Based on our client interactions and updates from various industry sources, some of the business problems introduced due to the pandemic in the manufacturing sector are:

Figure 1 Post-Pandemic Challenges Faced by the Global Manufacturing Sector

2. How Mixed Reality Can Cater to Some of the Business Problems?

Mixed Reality has enhanced holographic technology from being a mere mode of amusement, to a full-fledged interactive application for solving business problems. It has proved its caliber in various industries such as Education, Manufacturing, Medical, Real Estate and so on. Microsoft HoloLens further extends the holographic technology through integration with Microsoft Dynamics 365 Remote Support. Thus, making it a complete practical solution for solving problems remotely through the power of Holograms and remote coordination.

Figure 2 Mixed Reality, Augmented Reality and Virtual Reality

2.1 Factory Floor Supervision & Coordination Challenges

The pandemic brings along with it following challenges when it comes to factory floor supervision with limited staff:

Remote Operations is possible for back office and administrative work, where physical presence is not needed. On-site manufacturing activities are badly hit due to the need for physical presence and support.
Machinery maintenance in factories is a critical area where lower number of staff and increased amount of maintenance add to the workload and stress.
The Factory Foreman has to juggle between reports and maintenance logs while allocating work to limited staff of available technicians.
Technicians need to address more issues in shorter hours which leads to haste and chance of human errors.

2.2 Solution Derived Through Mixed Reality

Using Mixed Reality, these challenges can be addressed by stitching Mixed Reality with IoT dashboards, Smart Cameras (Computer Vision) and Microsoft Dynamics 365 Remote Assistance:

Through Mixed Reality connected with IoT, the Foreman can get real-time updates and alerts to inspect the affected areas of the factory and machines.
The Foreman can recreate the holographic representation of the issue wherever he/she is located (i.e., regardless of being within factory premise or at a remote location).
Through the data shared by the Smart Cameras (Computer Vision), the Foreman would be notified to check the area of non-compliance by workers.
Through the data captured by IoT devices connected to HoloLens, the Foreman would get instant notification of machinery faults to avoid accidents.

3. Use Case - Tracking Malfunction & Non-Compliance Using MR

Samuel is a Factory Foreman who is concerned with the supervision of people, machinery and safety compliance across the vast factory floor. Often, he feels stressed while executing the work due to:

Large number of inspections of machinery, tools, people and compliance
Frequent non-compliance issues due to staff carelessness
Huge manual paperwork involved in maintaining factory logs, job-sheets, maintenance reports, etc.

3.1 Samuel’s Goal
Remain stress-free from worries and human errors while executing his activities. To be so, he needs to be comfortable and capable of performing the job through the tools and technology provided to him, especially during the pandemic, where he falls short of people and time.

3.2 Using Mixed Reality to Assist Samuel for Meeting his Goals

Xoriant-Microsoft-HoloLens-Mixed-Reality-Tracking Malfunction-Non-Compliance-Manufacturing
Figure 3 Factory Floor Supervision using Mixed Reality

Samuel’s factory is enabled with IoT-powered smart machineries, devices and cameras, which send live feeds to the centralized IoT application server. The IoT Application server is configured to connect and synchronize with the IoT Client application installed on Microsoft HoloLens 2. This application accesses the dashboards, logs and reports processed by the Application Server based on the feeds received.

3.3 Samuel’s User Journey for Tracking Malfunction

The major challenge of the pandemic is the need for working remotely. While Samuel can visit a factory location, he occasionally would need to also coordinate remotely from his home or connect with other factory locations from the location he has visited.

3.3.1 User Journey for Tracking On-Premises Malfunction

Figure 4 illustrates Samuel’s journey of tracking malfunction using Microsoft HoloLens while being present within the factory premise.

Figure 4

3.3.2 User Journey for Tracking Malfunction Remotely

Figure 5 illustrates Samuel’s Journey of tracking malfunction using Microsoft HoloLens while being present remotely from within his home or another factory location.

Figure 5

3.4 Samuel’s User Journey for Tracking Non-Compliance

Apart from tracking malfunctions, Samuel is also responsible for ensuring the Safety and Compliance guidelines are followed appropriately and effectively for ensuring safe and healthy environment for the workers.

3.4.1 User Journey for Tracking On-Premises Compliance Issues

Figure 6 illustrates Samuel’s user journey for tracking compliance issues using HoloLens on the premise.

Figure 6

3.4.2 User Journey for Tracking Compliance Issues at Remote Location

Figure 7 illustrates Samuel’s user journey for tracking compliance issues using HoloLens from a remote location.

Figure 7

These use cases are few of those which can be applied to a Factory Foreman for accelerating his/her activities efficiently and effectively using Mixed Reality.

Many leading brands across the world are already leveraging HoloLens to provide real-time assistance to on-site operations for various functions. The alternative immersive tech solutions to transform manufacturing operations with wearable tech are still in different stages of development and not as advanced as HoloLens. The second generation of HoloLens was launched in 2021 with advanced features.

At Xoriant, we keep exploring the latest technologies to stay ahead of the times. Whether you want to develop apps using HoloLens or need a Mixed Reality solution to explore improved remote working possibilities for your workforce, you can connect with Xoriant experts.