PixelCastKit

Drawing on the Real World

I built a spatial drawing app for visionOS that lets users draw directly onto real-world surfaces using hand tracking or a Logitech Muse stylus.

The project started when I discovered that RealityKit's built-in pixelCast(from:to:) API wasn't available on visionOS—and took over 30ms to return a result in my testing. I saw an opportunity to build something faster, more flexible, and compatible with spatial computing.

Building PixelCastKit

I developed PixelCastKit, a custom ray casting library for RealityKit that uses Bounding Volume Hierarchy (BVH) acceleration to efficiently identify triangle intersections.

In my benchmarks, PixelCastKit returned results in under 1ms, delivering a 98.88% performance improvement over RealityKit's native method.

Beyond speed, PixelCastKit provides detailed intersection data, including UV coordinates, surface normals, world-space positions, and barycentric coordinates. This makes it possible to accurately map drawing input onto complex 3D surfaces.

Bringing It All Together

The app transforms ARKit plane detection and scene-reconstruction data into paintable RealityKit geometry. Users can then draw naturally onto their surroundings, with strokes rendered directly onto surface textures in real time.

The implementation combines several technologies:

• BVH-accelerated ray casting for fast, precise surface intersections

• RealityKit ECS systems for frame-by-frame input processing

• Metal-backed LowLevelTexture updates for real-time GPU texture painting

• ARKit plane detection and scene reconstruction for mapping physical environments

• Hand tracking and Logitech Muse stylus support for spatial drawing input

• PencilKit for familiar drawing tools and controls

Sharing the Technology

I also created Accelerated Pixel Casting in RealityKit, a course that explores the implementation and architecture behind the project.

It includes three reusable Swift packages—PixelCastKit, RKMesh, and SpatialAccessory—that developers can use to build their own spatial interactions, from texture painting and decals to precision selection and surface annotation.

iOS

I have also developed a working iOS version of the drawing app (as seen in this post's media), with plans for a future App Store release.