Unfortunately, the second part of this series was delayed as I healed from several injuries to my dominant arm. Now that I’m on the road to recovery and will be presenting some of this work at the 2023 HASTAC conference, I’m able to share this next part with you.

As a reminder from the last post, in the Spring of 2022, I had the opportunity to start work on a pilot project creating interactive experiences of structures within the Delaware Water Gap National Recreation Area combining augmented reality (AR) and soundscapes. This initial work uses a combination of 3D renderings of structures created using the photogrammetry mode of the Polycam app and sounds captured with an iPhone to create interactive experiences using Adobe Aero. In part one, I gave an introduction to the project and talked about the photogrammetry portion of the project. In this post, I'm going to cover the sound capture and the Adobe Aero portions of the project. For the purposes of this post, when I use the phrase “digital asset or “asset,” I am referring to things like audio files, photogrammetric scans, photos, or video. When talking about my phone and my iOS experiences, I am referring to an iPhone 11 Pro.

Soundscapes
Initially, I had wanted to capture soundscapes in situ at the structures using a Zoom handheld recorder. While this may have provided higher sound quality, I ultimately chose to record via iPhone voice memos to explore the ability to create AR experiences with the least amount of equipment possible. By using only the phone, I eliminate the need to carry extra equipment, which has implications for a variety of conditions under which one might want to collect digital assets, such as backcountry hiking/exploration. From a changing ecosystem and park use perspective, one soundscape for an experience may not be enough to reflect the diversity of the land as the seasons and conditions change. For example, the amount of visitors at the park, hunting seasons and shooting ranges on lands near the park, and mating seasons will all impact the collected soundscape. 

Adobe Aero
I chose Adobe Aero for this project for several reasons, chief among them the barrier to entry for someone new to AR. It is free, there exist tutorials, and it does not require the steep learning curve and resources that Unity would have. I also was drawn to Aero because it allows me to link audio and visual assets through the Behaviors settings. It’s important to note that Adobe Aero is available for iOS, but is in beta release for Mac and Windows desktop. While it is possible to create an entire experience from asset collection to processing to constructing and deploying the AR experience on iOS with the app, I personally find it much easier to collect my assets and then work with the beta desktop version for Mac (for me, a MacBook Pro 2017 (RIP), MacBook Pro 2023). 

The workflow that works best for me is to capture my assets (soundscapes and photogrammetry) in situ on the phone at my site of interest and then work from them at home on the desktop version of Aero.*

First, I prep the assets. For the 3D model, I use a .glb file from Polycam. With a pro subscription, Polycam provides users with several different export types. Determining which would work with Adobe Aero required experimentation, I ultimately found that the .glb file export was the option that worked for me. On the free/trial version of Polycam, .glb is the only export option. For the audio files, because the sound recordings on an iPhone save as .m4a files, I convert them on desktop to .wav versions in Apple Music and then use the .wav within Aero.

To build the experience in Aero, I start a new experience and add the .glb file. I rotate and resize the model and place it in coordinate space as I would like it to appear. Then I open the Behavior menu in Aero, add a Proximity Entrance trigger, choose the model as the subject, select the distance at which it activates, and decide on the re-triggering criteria. I add a play audio action, choose the subject to be the model, upload my audio file, and adjust the play count if needed. From there, I am ready to test my AR experience.

While Adobe Aero was an interesting tool for this pilot; it is not without limitations. First, Aero is only available to iOS users and requires a Creative Cloud account to view the experiences. This drastically lowers the availability of the experience to potential audiences. Second, I’ve recently had cross-platform issues when working on an experience: if I made edits in the app version, I could no longer access the experience in the desktop version.

Example: Lower Treible House (A structure at Peters Valley School of Craft)
Below is test footage of an AR experience created in Adobe Aero using a model built on a photogrammetric scan of the Lower Treible House from Polycam combined with audio recorded next to the structure in the parking lot on a different day (same week) as a voice memo. The audio is proximity entrance triggered. I’m beta testing this app in a hotel room. (Updated on 6/9/2023 with new example).

Closing Notes
Ultimately, I consider my pilot to create AR experiences quickly with minimal equipment a success. However, I do believe that combining LiDAR scanning with drones for the photogrammetry and specific recording equipment for outdoor soundscape capture would yield a higher quality augmented reality experience. However, for beta tests and pilot projects, my iPhone and MacBookPro were enough for me to create an interactive experience quickly and cheaply. In an ideal case, I would love to design these experiences on a platform that is free, operating system agnostic, and does not require an additional log-in step.

———————————————————————————————————-

*I have found that you can to create the whole experience on an iPhone by saving the .glb to your phone and doing the audio file type conversion in GarageBand It may take a little bit longer depending on the size of the glb file.