Welcome back! In this three-part blog series, I explore marker and image tracking in XR to untangle some of the confusion around these terms, highlight some success stories and finally discuss the future of marker and image tracking from a web standardisation point of view. The series will cover the following in 3 separate posts:
Before we kickstart this second part, I had a couple of quick disclaimers to make:
- I used a mix of terms to describe different types of tracking in the first part of this series and that may add to the very confusion that I am trying to untangle. To clarify — “marker and image tracking” refers to fiducial marker tracking and image tracking, whereas “marker based experiences” or “mobile based AR experiences” refers to both.
- The title of the series does have “on the Web” in it because that’s the main focus, but important to note that many of the topics covered here are also applicable to native applications.
In the first blog post in this series, I zoomed into a few historical milestones to untangle some of the confusion around the correct terms for these technologies, and better understand where they came from. In this second part I am keen to highlight some success stories in using marker and image tracking and also briefly reflect on my own personal experience in developing marker based WebXR experiences.
Before I jump into specific use cases and examples however, I wanted to firstly emphasise the growing trend of marker and image based experiences from a commercial point of view. Besides ARCore and ARKit, and in addition to academic efforts and great open source solutions mentioned in the previous post in this series like ARToolKit, every other major provider of XR products and services like Niantic, PTC, Snap, TeamViewer, Vuzix, Varjo, Magic Leap (and many others) are integrating marker and image tracking into their offerings. To me this screams the following: marker and image tracking is still at the forefront of XR development. There is high and growing demand for these experiences and accordingly leading XR companies are making substantial investments to support and deliver high quality marker and image tracking experiences.
At this point you may be thinking “well… that’s great but surely that only benefits those enterprises?!”, and that’s a fair point but here is something to keep in mind— many of these companies are not only pushing out products and frameworks, but they are also helping shape the future of XR through active participation in relevant W3C working groups and other efforts to establish open standards that will ensure these technologies continue to grow and improve for end users and developers. By providing these solutions and pushing the boundaries of what’s possible in marker and image tracking, they’re supporting not just their own innovation, but the entire XR ecosystem including researchers and small developers.
Specific successes
There are many individual success stories throughout history when it comes to marker and image tracking experiences and it would take more than one post to cover them, but here are a few that I came across over the past few years that I particularly liked:
Notre-Dame de Paris: The Augmented Exhibition
French start-up Histovery has recently developed a unique marker based AR experience to explore Notre-Dame’s heritage which attracted nearly half a million visitors across different cities worldwide such as Paris, Washington, Montreal, Dubai and London. Powered by Samsung’s new Galaxy Tab Active5, visitors are able to scan different physical scenography (i.e. trained image markers) and launch immersive experiences to relive several key historical moments from its construction at different points in time. See a video trailer for Notre Dame de Paris, the Augmented Exhibition at Westminster Abbey: trailer.
Learn more about Notre-Dame de Paris: The Augmented Exhibition.
Merge Cube
This is something that I came across very recently, and it provides a physical cube which has unique markers on each side of the cube that can be scanned to overlay virtual content on any given face (different markers for different content of course). They also provide an optional cardboard like viewer for more immersive viewing of virtual content. While it does require installation of a dedicated app to build and view experiences (so not entirely web based), I found this super interesting because it simplifies marker and image tracking in a way that works very well for education purposes and the overall design and execution are very welcoming for students and education related activities. See a video of this in action for How does the Merge Cube work?.
Learn more about Merge Cube
L’Oréal
I personally almost use virtual try-ons as a great example for effective mobile based AR experiences, and L’Oréal in particular have been super successful in using this technology effectively (fun fact: L’Oréal were also the sponsors of the Notre-Dame de Paris: The Augmented Exhibition). Important to note here that virtual try-ons can deviate a little from marker and image based tracking and lean more towards marker-less AR or face/body tracking (where your face/body/hand/hair is essentially the marker), but the point still stands that these tracking experiences are mostly mobile based and require user actions to activate and use. You can watch a video of this in action for Virtually Try On Makeup (notice how old this video is, L’Oréal have been all over this technology since day one).
Learn more about Virtual Beauty Try On Tools
LEGO
The infamous toy manufacturing company has long embraced marker based AR experiences to reimagine how users interact with their products, and extend the fun aspect beyond the physical packages of their toys. Through dedicated apps, users are able to scan a variety of physical features (e.g. images, physical objects etc.) to launch engaging immersive experiences and interact with them. Watch an example of one of those experiences for Merlok 2.0 — Lego Nexo Knights.
Learn more about LEGO® Technic™ AR App
Zappar
This example has more to do with QR codes that actual AR markers, but I absolutely had to include it here under the umbrella of “mobile-based experiences” because of how valuable it is. Zappar is one of the leading mobile based XR companies out there, and they recently introduced the concept of Accessible QR Codes (AQR) for people who are blind or have low vision. AQR are essentially enhanced QR codes which, once scanned, announce information related to a product or signage using the device’s accessibility features (e.g. text-to-speech, larger fonts) and greatly enhance the shopping experience for those user groups which is pretty cool. Another reason why I included this particular example is to illustrate that innovations coming from XR companies go beyond commercial success, and does in most cases provide valuable accessible tools and solutions to all users.
Learn more about Zapvision: What is Accessible QR (AQR)?.
A short personal tale
I had the pleasure of working on a project that investigated the use of AR for consumer healthcare applications in a previous role, and for that I needed to create numerous WebXR experiences, many of which were marker based. Open source solutions were my greatest ally during this period because they were accessible, had great documentation and samples to get started. In hindsight, the tracking of markers could have been better, but all in all these tools allowed me to deliver exactly what I, and the project, needed. They perfectly showcased the potential of marker and image tracking and were great precursors for scaled up experiences using premium solutions. I wish I can share some of the demos I built, but none of them belong to me as far as I am aware, so I will have to settle for a shameless plug for a talk I gave about the project which should give you a rough idea of how we used marker-based AR in a consumer healthcare context.
When I was proof reading this post and looking at all of the examples above, I found the spectrum of applications that marker and image tracking is used for quite interesting. Many may argue that most marker based experiences are purely for entertainment and marketing campaigns, but those experiences can indeed also be utilised in other domains (e.g. consumer health, remote assistance, accessibility and many others).
Based on the examples above and my own personal experience as a developer, I usually still find that marker and image tracking experiences can divide opinions. Here is my take on some of the perceived disadvantages of these experiences from a developer and user points of view:
Not truly immersive
Marker based XR experiences can be deemed as “not immersive enough” if an XR headset is not being used by the user to view and engage with the experience. This particular point presents the perfect opportunity to reiterate a point I made in a previous blog post — immersion is inherently subjective, and a low level of immersion to one user might very well feel like the perfect level of immersion to another. Marker based XR experiences offer a different level of immersion which is not necessarily tied to a headset, but they still offer much of the elements that make up an immersive experience (e.g. interactivity, graphics, real world understanding). Completely discarding marker based tracking for this reason risks excluding users that can only use and access XR experiences on non-headset devices.
Ergonomics of handheld devices don’t help
Holding a mobile device in awkward positions for prolonged periods of time can be viewed as a significant barrier for adopting handheld AR experiences. My problem with this argument is that it ties the value of handheld AR (and the support for it) to mass adoption and traction or to a killer app that leads everyday users to interface with mobile AR 24/7. My view is, handheld AR doesn’t need to be used continuously or for long periods of time to be valuable. The beauty of handheld AR lies in its ability to provide quick, context-specific interactions and experiences. Whether it’s helping someone visualise how a piece of furniture fits in their living room or providing instant product information, the value is in its ability to support real-world decisions, not necessarily in how long or how often the user engages with it.
Marker based experiences are too basic
Closely linked to the first point, it is often perceived that marker based experiences only deliver basic immersive information (for example — a single 3D model, a single animation or very basic interactivity), but the reality is that it should be technically possible to deliver any supported feature by the WebXR Device API via a marker once the experience is activated (with the potential limitation of the marker needing to be in the view of the camera throughout that experience).
I personally tend to subscribe to the opinion that marker and image based experiences are valuable, and it would only be fair to highlight some of the perceived advantages of marker based experiences as well — so here goes my take on those:
Low barriers to entry
There are virtually no barriers to entry when it comes to using markers or scanning QR codes. Anyone with a smartphone or/and a tablet and an internet connection can access an XR experience on the web. It really is as simple as that.
Makes use of everyday technology
I wanted to add some statistics here about the amount of smartphones and smartphone users in the UK but do I really need to? Marker based experiences make perfect use of everyday technology that is — already — widely accepted by users and society in general.
Accessible to everyday users
Marker and image tracking works really well for everyday users because the instant access advantage suits their daily contexts when using immersive technologies. Someone who can only spend 2–3 minutes in a shopping trip would happily scan a marker to access an immersive experience that offers them something in return, because they know it won’t take long to have that experience and reward. They simply wouldn’t use XR if they were asked for additional steps to get that access during their limited time context (e.g. install an app, make a payment etc.).
Success stories
As mentioned above, there are many success stories in using marker and image tracking for helping consumers make better decisions (again decisions that they make in specific contexts). Virtual try-ons, product activations and unique marketing campaigns. These success stories are not just about big businesses making money, but also XR development companies that have excelled at using marker and image based tracking to provide unique experiences for end customers and users over the web and natively.
And that’s that for success stories and some perceived pros and cons of marker and image tracking in XR and WebXR! I hope this was useful in highlighting the current use of these technologies and their huge potential. But what does the future hold and how can you get started today? To answer these questions I will be putting my Web Standards hat on in the next and final part of this series to discuss the current status of marker and image tracking from a web standardisation point of view, and highlight the tools developers can use today to achieve robust marker and image tracking XR experiences. Stay tuned!
Muadh out — Until next time! 🤜🤛
