An Immersive AR Art Experience

An augmented reality art viewer built with Swift and RealityKit, allowing users to discover, collect, and experience digital artwork in their physical surroundings.

The experience combined custom rendering techniques, intuitive spatial interactions, and location-based discovery to make viewing digital art more immersive and interactive.

The Experience

Users could explore a collection of digital artworks, switching seamlessly between AR and non-AR viewing modes. In AR, artworks could be positioned throughout the physical environment, transforming walls and ceilings into virtual gallery spaces.

The experience was available through both a full iOS app and an App Clip, allowing users to scan a QR code and jump directly into the artwork without installing the full application. Users with the app already installed could launch the full experience instead.

Location-based discovery added another dimension to the experience, allowing users to visit specific physical locations to unlock exclusive artwork skins.

My Role — Lead iOS Engineer

As Lead iOS Engineer, I was responsible for much of the core RealityKit implementation, with a particular focus on custom AR interactions, shader development, rendering techniques, spatial audio, haptics, and photo and video capture.

REST API and Artwork Downloads

I used HTTP networking with a REST API through the Alamofire Swift package to download 3D artworks, load which pieces a user owned and which locations they had unlocked, and post new location unlocks.

Custom Spatial Interactions

One of the most technically interesting challenges was building a custom gesture system inspired by Apple's AR Quick Look experience.

Rather than relying on RealityKit's default gesture recognizers, I implemented custom interactions that gave us greater control over how users manipulated artwork in AR.

Users could drag artwork naturally around a room, with the artwork dynamically aligning to surrounding surfaces. As it moved from a wall to the ceiling, it could smoothly reorient itself—even flipping upside down to maintain the correct placement.

Building our own interaction system also gave us the flexibility to integrate custom rendering effects and extend the experience beyond what AR Quick Look supported at the time.

Custom Shaders and Materials

I designed and implemented a custom iridescence shader, creating a dynamic, color-shifting appearance that responded to the viewing angle. The project also featured artistic procedural patterns and other custom material effects that gave the digital artwork a distinctive visual identity.

Another technical accomplishment was developing a custom lit video material in RealityKit. Apple's built-in video material was unlit, meaning it didn't respond to scene lighting. I found a way to create a video-based material that could participate in lighting calculations, opening up new possibilities for integrating animated artwork into AR environments.

Particles

The particle effect seen in the video was a custom effect I built using instancing of billboarded planes. This is because at the time of the recording, RealityKit did not yet have a built in particle system (it provided one to iOS in iOS 18).

Spatial Audio

I also built the spatial audio system that placed each artwork’s sound in the room. RealityKit playback went through a custom component and per-frame system. An AVAudioEnvironmentNode treated the camera as the listener, updating its position and orientation every frame, while each emitter’s player node followed the artwork in world space.

Tracks could be spatial, ambient, or non-spatial. Layered clips started together on a shared audio timeline so they stayed in sync. Playback supported looping, pause and resume from the last sample, fade in and fade out, and room reverb presets from a small room to a cathedral. A parallel SceneKit path attached positional sources to the artwork, with distance rolloff and per-source reverb. When someone recorded the experience, a tap on the engine mixer captured that spatial mix, so the recording included the artwork audio along with the microphone.

Haptics

Haptics marked the moments when an artwork settled into place. A soft impact fired when a piece finished anchoring, when a pinch hit the minimum or maximum scale, and when the scale snapped back to 100%. The same bump confirmed a photo or recording, and the shutter used selection feedback as the capture controls changed. Those generators were prepared ahead of the gesture so the hit landed with the action instead of a frame later.

For longer moments I used Core Haptics. A CHHapticEngine played designed patterns: a continuous low rumble that holds and then fades, layered with sharp transient sparkles, plus shorter slice, crunch, and splash patterns built from intensity and sharpness curves. The engine restarts itself if the system resets it, and the audio session keeps haptics enabled while a recording is in progress so the Taptic Engine still responds during capture.

Photos, Video, and Sharing

People could photograph the AR scene or record it. A snapshot grabbed the current RealityKit or SceneKit frame and, in the full app, saved it to the camera roll. Video recorded the rendered view as H.264 MP4 at the screen’s pixel size and 60 frames per second. When the artwork had spatial audio, a tap on the audio engine wrote that mix into the file. Otherwise the microphone was recorded separately and merged onto the video with AVFoundation before the clip was saved.

Sharing used the system share sheet. The activity items carried the image or video plus the creator, title, edition, and a link back to the piece. On iPad the sheet was anchored as a popover. The App Clip could not write to the camera roll, so a capture opened the share sheet immediately, and every few captures it offered the full app so people could save recordings there.

Technical Highlights

• Swift & RealityKit — Native iOS augmented reality development

• App Clips & QR Codes — Instant, installation-free entry into the AR experience

• Custom AR Gestures — Surface-aware artwork placement across walls and ceilings

• Custom Shaders — Iridescence and artistic digital material effects

• Lit Video Materials — Custom rendering that enabled video textures to respond to lighting

• Location-Based Unlocks — Geographically triggered artwork skins

• AR & Non-AR Viewing — Multiple ways to explore and interact with artwork

• Artwork Collections — Browsing and viewing multiple digital artworks within a single app

• Alamofire & REST — Downloaded 3D artworks, read owned pieces and unlocked locations, and posted location unlocks

• Spatial Audio — Camera-tracked 3D audio locked to each artwork, with synced layers, fades, reverb, and recordings that include the mix

• Haptics — Soft impacts for scale snaps, anchoring, and capture, plus Core Haptics patterns for longer transitions

• Capture and Sharing — AR photos and 60 fps H.264 video, with spatial audio or a merged microphone track, shared with the artwork title and link