Listening to a sculpture by Henry Moore

In the following video, you can listen to various positions around a sculpture by Henry Moore. Each position is played 3 times. The sound is triggered with a pre-recorded sample of snapping fingers.

When you compare the sonification with the 3D mesh in the video, you will find that the louder sounds belong to the brighter surface triangles or areas of the sculpture closer to and facing you.

Pilot study: testing proof of concept

In September of 2012 we had the opportunity to test our project with various participants from the visually impaired community in Montreal.  It was exciting to see that some of our audio rendering cues made sense to them and that our participants were able to identify the different positions of the sculptures tested.  Big thank you to all of our participants for kindly coming to CIRMMT to test our proof of concept.

Transient

For our pilot study we built small replicas of various existing sculptures; some of them found in the Mont Royal park.  We placed the replicas on a rotating knob that allowed our participants to explore them from different view points (by rotating the knob).  This was analogous to going around the sculpture.  The exploration consisted of touching the sculpture and snapping fingers (or tongue clicking) from each different position.  

The sculpture replied back to each sound produced by the participant. The audible response was created by convolving the input sounds with a sonification of the sculpture that was based on the triangular mesh from the 3D model.  We took into account the triangle size, its vertical position and the surface-normals with respect to the listener perspective.  The interaction between the participant and the sculpture is illustrated in the video below.

Comparing two audio rendering approaches

The example illustrated here relates to a sculpture by Henri Moore.

The following audio files present an example of the two different audio rendering approaches tested. The first one, shockwave, is inspired by echolocation, where surface triangles respond according to their distance to the listener. 

The second approach, panorama, is a scan across the stereo-panorama from left to right.

Transient
Transient

From stone to sound

The diagram below presents our pipeline to create the audible sculptures.  It goes from measuring the sculptures to generating a 3D model of the sculpture, all the way to our real time sound synthesis program in SuperCollider.  

Using the OBJ format was the bridge between the 3D world and our audio synthesis program.  We found the description of the OBJ format by Paul Bourke particularly useful. Thanks to the the Wavefront OBJ Matlab toolbox by Dirk-Jan Kroo we were able to import our OBJ files into Matlab.  From here it was easy to manipulate a large matrix with the vectors and surface normals to then output a CVS file with a data format that facilitated the reading and processing of the 3D model in SuperCollider.