IdeAR: Enhancing University Education with Augmented Reality, Part 2

In the previous article, we talked about the origins of IdeAR, explored its standout features and delved into the integration of React and BabylonJS. If you happened to miss it, catchup by following this link: https://futured.deakin.edu.au/idear-enhancing-university-education-with-augmented-reality-part-1/.

Why Angular

In modernising IdeAR, I was tasked with bringing it up to speed with the latest package versions of both React and BabylonJS. Given the intricate routing complexities within the IdeAR app, a strategic decision was made to transition to Angular. This move not only aligns IdeAR with the latest industry standards but also ensures consistency with other applications under development in the Software Engineering domain at Deakin.

Modernising IdeAR:

Step 1 – Rendering a model

The first step in the transition to Angular involved creating a proof of concept (POC). This POC was a simple Angular application incorporating the latest BabylonJS packages, designed to render a single prepared glTF 3D model. I discovered this great online template, which I used as my base to start developing.

The template contains the base code for creating your engine, scene and camera as well as loading a model. However, I found it necessary to enhance the default camera settings so that the model displayed correctly within the canvas, facing the appropriate angle, and restricting the user from zooming too far in or out. Getting the desired angles proved to be quite complex with the setPosition() method.

After copying the glTF models from the current app, the engine was updated to load the 3D model. And now, for the pièce de résistance.

Step 2 – Handling click events on model layers

Now that I have a working app displaying a 3D model, my goal is to replicate the functionality provided by the React app. This means making the model interactive, allowing it to highlight the selected layer, and providing a description for the selected area.

Firstly, I need to create a highlight layer. An EventListener is added to the canvas to handle when the mouse button is clicked. If the mouse button is clicked on a particular model layer using “myScene.pick(myScene.pointerX, myScene.pointerY)” the current highlighted layer is cleared, and the selected layer is then highlighted.

Step 3 – Displaying model and layer details

Next, I needed to add the name of the selected layer at the top of the model.  Firstly, we create an AdvancedDynamicTexture and add a TextBlock to it. The text could be set to the pickedMesh.name value, but we have an API that retrieves better descriptions from a database store.

Now, the left-hand sidebar needs to be updated to display the model name, layer name and any associated description for the selected mesh. To enable this, we can create a DataSharingService class.

In our Babylon engine, we can now inject a DataSharingService into the constructor and push the details (eg. layerName) when the mesh is clicked, by using the .next() method.

   

In the component where we display the selected mesh text, we need to inject the DataSharingService and subscribe to the attribute changes. Then in the related html we can display the layername.

   

I have done this to include the model name, model description, and any available layer descriptions, resulting in the following:

Step 4 – Adding toolbars

Next, we want to complete the display by adding both vertical and horizontal toolbars to the bottom right section. These toolbars will handle zooming, panning, resetting and more. First, we should create the toolbar in a new component and add this to the app.component.html.  I’ve only shown one button here, but you can add more buttons such as reset, move model up, move model down, zoom in and zoom out.

With our buttons in place, the next step is to manage their clicks. In order to achieve this, we’ll integrate the previously crafted DataSharingService into the component and send a message for each button click. Note that the DataSharingService will need to be updated with each of the button press variables (boolean).

The engine service now needs to be updated to subscribe to the button presses and perform the appropriate action:

   

Repeat the same process for the horizontal toolbar. I have added the CSS and actions below to guide you through the process.

Finally, this is how the model with the toolbars should look.

Current Progress and Future Goals

In the ongoing transition of IdeAR to Angular, we’ve successfully achieved the display of 3D models and their various parts. However, the next crucial milestone involves seamless integration of the annotation feature seamlessly within the Angular environment. This involves reimagining the user interface and interactions to ensure a consistent and intuitive user-friendly experience. Our goal is to create a powerful and user-friendly AR tool that continues to revolutionize university education by making complex subjects more approachable and engaging for students.

Conclusion

IdeAR stands out as a shining testament to the transformative power of augmented reality in the educational landscape. Its journey from React to Angular, coupled with the latest BabylonJS libraries, promises an even more immersive and interactive learning environment for both educators and students. As the project progresses, IdeAR has the potential to redefine how we approach complex subjects in the digital age. Stay tuned for the next chapters in the exciting narrative of IdeAR’s technological prowess and educational impact.