How can extended reality technologies ‘awake’ the untold stories and generational knowledge of Australia’s First Nations people?

From the first time University of Melbourne anthologist Donald Thomson set out on horseback for Far North Queensland in 1928, he captured life in First Nations communities across Australia with a natural curiosity.

Over the course of his 50-year career, he collected thousands of First Nations belongings while living in remote communities in Cape York Peninsula, Arnhem Land and the Western and Central Deserts, securing his place as one of the 20th century’s foremost anthropologists.

The belongings are now part of the Donald Thomson collection, recognised in the UNESCO Australian Memory of the World register and presenting rich, multi-faceted insights into the living culture of Australia’s First Nations people.

Exploring the relationship between people, place and content is central to all projects we take on at Art Processors. So when the University of Melbourne’s Faculty of Arts invited us to “awaken” belongings from the collection for the aptly titled Awaken exhibition, we wanted to create an accessible technological solution that blended the physical and digital worlds.

Just as Thomson used the latest technologies available to him at the time — he was one of the first filmmakers to shoot 16mm colour film in Australia — we sought to push the boundaries of extended reality technologies available to us today.

Challenge

Genevieve Grieves, exhibition curator and Melbourne Museum’s Director of First Peoples, had an overarching vision for Awaken that focused on inclusion and community. The exhibition was more than just a window to look back at history with fixed perspectives — it was an opportunity to move forward; to reconnect the belongings with community and to start new conversations around future outcomes for First Peoples.

Approach

Working closely with the University’s Faculty of Arts along with the Digitisation Centre and NExT Lab, we explored various approaches to the 3D scanning of the belongings. With the help of The Mulka Project and representatives from the remote communities, we captured 360º audio-visuals of the natural landscapes where Thomson originally sourced the belongings.

Deliverables

An immersive VR experience that transports visitors to remote parts of Queensland and the Northern Territory where digital surrogates of the belongings are “re-placed” in their original landscapes. An AR experience that “unlocks” the belongings from their glass cabinets, for an interactive and hands-on experience. Digital labels that deliver rich information, including extended descriptions and other media.

“[This experience] has been incredibly rewarding for me, as I was able to reconnect with my ancestral materials and further inform the families back home about the project and collection… Many of the objects had not yet been seen by families, so the consultation period gave mob an opportunity to see the objects and also hear about the extensive Donald Thomson Collection.”
Shonae Hobson, Curator, Awaken Exhibition

A common resolve

“Art Processors understood from the outset our objectives to reconnect the objects with community through a contemporary, best practice approach that would complement the Arts West object-engaged learning methodology. They delivered an innovative solution that opens up new ways of cultural exchange, of communication and of exploring our shared history.”
Professor Russell Goulbourne, Dean, Faculty of Arts, University of Melbourne

The Donald Thomson Collection is vast. It spans anthropology and botany, through to zoology and studies of Aboriginal ecology. For sheer quantity and diversity, it ranks amongst the most important anthropological collections in the world. But for over 80 years, it lay largely dormant in storage.

Working with Melbourne Museum curator Rosemary Wrench (who was the principal consultant for the film Ten Canoes (2016)), The Mulka Project, Southern Kaantju woman and Faculty of Arts alumna Shonae Hobson, together with representatives and elders from remote communities, there was a common resolve from all involved to explore and redefine the enduring traces of the collection.

We were tasked with “awakening” several of the belongings. Specifically, to activate the experience of the belongings in relation to their present dislocation with country, with the goal of restoring their context and connection to First Nations knowledge.

How do you awaken inanimate objects? And, in this case, unlock the extraordinary knowledge embedded in them?

Thomson’s immersion in remote communities inspired us to explore a contemporary approach that applied object-based learning, the Faculty of Arts’ vision for teaching that involves integrating authentic or replica material objects into the learning environment for a more interactive, hands-on experience.

Representatives from the Yolngu, Cape York and Pintupi communities spoke of their personal links to the belongings as generational landmarks; vessels for passing on generational knowledge embedded in their creation.

With this in mind, we saw our opportunity for object-based learning experiences — the creation of digital surrogates that visitors could interact with physically and digitally via VR and AR.

Creating the digital surrogates

“For the 3D digitisation process, we worked with the University of Melbourne’s Digitisation Centre, The Faculty of Arts’ Digital Studio and NExT Lab to explore various approaches to scanning on-site at Museums Victoria’s Collection Store. This process included trials of structured light scanning with blue lasers and photogrammetry techniques to capture the form of the objects and their surfaces. Over the course of several weeks, we iterated on the workflow needed to produce high-resolution 3D meshes (20M polygon count), physically based rendering (PBR) materials and textures for the belongings’ virtual presentation.”
Chuan Lim, Creative Technologist, Art Processors
Modeling credit Chuan Lim and Ben Kreunen

Capturing connection to country

Melbourne, where the belongings are now located, is a world away from the communities where Thomson was immersed. We’re talking vast distances, some of the most remote parts of Australia.

We wanted to capture these locations on film, but we needed help.

The Mulka Project, an organisation whose mission is to sustain and protect Yolŋu cultural knowledge, was instrumental in filming the landscape of Northeast Arnhem Land. Using stereoscopic 3D cameras, they captured verdant “mayang”, or inland rivers, and desolate sandy beaches.

For the communities that didn’t have their own equipment, we provided Ricoh Theta 360° cameras, tripods and spatial microphones, together with hands-on training. Multi-directional microphones attached to the camera setups enabled field recordings of the location ambience in 3D as first-order ambisonics.

When processed inside of a real-time game engine and then played back through a two-channel binaural output, we were able to recreate the experience of the sound as heard on-location, responsive to head angle and rotation, presenting the rich, aural texture of the water, wind and wildlife specific to those regional ecologies.

Re-placing the belongings with VR

“Re-placing the belongings was more than just a technical achievement. We created a context-based experience that synchronised time and place, sound and vision, awakening old and new connections between the belongings, the communities, the country, and those discovering the collection.”
Tony Holzner, Creative Director and Co-Founder, Art Processors

If the landscapes are vital to understanding the history and knowledge embedded in the creation of the belongings, why not take people there?

Using sunlight captured in the 360º video filmed in the three remote regions, we were able to re-light the digital surrogates, in effect reuniting them with their original landscapes for the first time in nearly a century.

Technically, this involved taking the lighting conditions in the environment maps and filtering them through physically-based rendering (PBR) materials to create a sense of the natural landscapes illuminating and interacting with the dormant objects.

Virtual reality and Oculus Go headsets provided the means for transporting visitors to the far-flung reaches of the Northern Territory and Queensland where they could see and hear the belongings in their remote environments.

Getting hands-on with AR

To complement the experience of going inside the VR environments, we used augmented reality to “unlock” the belongings from behind their glass cabinets.

Using AR Kit, the digital surrogates can be displayed onto a large communal table with AR object markers in the middle of the room. The round table encourages group interaction and discussion as visitors gather around for a shared experience learning about the history and tradition of the belongings.

These tangible object markers — with 2D feature tracking — enable hands-on, physical interaction with the 3D models that are projected onto them, allowing visitors to pick up the digital surrogates as seen through the AR frame, while also reconnecting the objects with community.

This approach has been successful, providing a seamless experience centred firmly on the objects at hand, while at the same time revealing more about the collection than traditional fixed-label interpretation. It also highlights the kind of open-ended user experiences that mixed technologies offer when we provide “as little design as possible” — and are willing to trust an audience’s instinct for discovery and exploration.

Developing digital labels

In lieu of static wall labels inside the display cabinets, we developed digital labels that encourage visitors to engage with the physical objects on display. We felt it was important to provide some context to the practical use and function of the belongings, including the materials used in their construction and First Nations knowledge of the natural environment.

The digital labels prioritise the immediacy and wonder of engaging with the belongings in the collection before encountering secondary sources of information on 10” iPad Pro tablets. The tablets serve as portable guides, delivering a breadth of rich information, and drilling down into extended descriptions, additional curator’s notes, and other media.

Awakening untold stories together

One thing’s for sure: Awaken is the culmination of a truly collaborative process that brought together many passionate and generous people from all over Australia.

This project was perhaps the most complex we’ve worked on in terms of the number of stakeholders involved — institutions, museums, remote communities and community groups came together, often from vast distances, to contribute their skills and knowledge, ultimately realising the original goals of Awaken.

It was humbling to be part of this experience and to collaborate directly with First Nations communities so their generational knowledge and stories could be told, and the Donald Thomson Collection could be made accessible to remote communities.

Given the change over time in cultural heritage management, communities of origin are interested in negotiating the return of their cultural objects. Perhaps there is a role for both real and virtual belongings to play in a return, providing a link between past and present and to maintain connections vital to these communities’ continuing story and identity. This is an ongoing conversation, and one we encourage.