Cécile Picard


Retargetting Example Sounds
to Interactive Physics-Driven Animations


We present the results of our method to generate audio in the context of interactive animations driven by a physics engine, such as those used extensively in games. Our approach consists in retargetting audio grains extracted from pre-recorded contact sounds to interactive physics-driven animations.

We first present results of our approach for resynthesis. In this case, the purpose is simply to reduce the size of the recording database manipulated during real-time interaction in video games. The method is applied on recorded contact events involving different material types (e.g. stone glass, metal, wood, etc..) and interaction types (e.g. break, hit, roll, slide). We also demonstrate that our technique is well adapted for the specific case of continuous-like events such as rolling or sliding. A specific operation is performed on continuous-like recordings that separates the sinusoidal and the transient parts to allow coding them separately.

Using this approach, the audio grains can be efficiently concatenated using the coding structure of user-defined, shader-like procedures. Similar audio events can be built using different approaches. Time-scale modification of the original recordings is easily implemented acting on the Time Modification factor (TM); TM<1 induces shrinking whereas TM>1 induces stretching. Different examples are given (1). Besides time scaling, continuous-like recordings are varied by enhancing the sinusoidal part or the transient part extracted in a preprocess analysis step, acting on the granularity sensation of the contact interaction (2). Finally, variable audio content is appropriately matched to a caracteristic rhythmic pattern of an available audio recording using cross-correlations calculations(3).

Finally, we present our results for an interactive video sequence, similar to a game setting. Here atom distribution is tailored by parameters from game data. Sounding material is retargeted to simulated sounding events through the amplitude of the normal force and relative velocity parameters.

Each sound clip contains the original exemplar followed by the synthesis result.

Resynthesis Results

STONE Interactions

Total Resynthesis Encoding Size (vs original) = 80%
   IMPACT-like contacts
     . Stone/Break (30 records): 1034 impulsive atoms
       Stone/Break events     1     2     3
     . Stone/Hit (44 records): 1086 impulsive atoms
       Stone/Hit events     1     2     3
   CONTINUOUS-like contacts
     . Stone/Roll (3 records): 276 impulsive atoms - 313 continuous atoms
       Stone/Roll events     1     2
     . Stone/Slide (17 records): 1250 impulsive atoms - 1212 continuous atoms
       Stone/Slide events     1     2          Stone/Gun events     1     2

WOOD Interactions

   IMPACT-like contacts
     . Wood/Break events     1     2          . Wood/Hit events     1    2
   CONTINUOUS-like contacts
     . Wood/Roll events     1     2     3     4

GLASS Interactions

   IMPACT-like contacts
     . Glass/Break events     1     2          . Glass/Hit     1    2
   CONTINUOUS-like contacts
     . Glass/Roll     1     2     3     4

Results of Similar Patterns Synthesis

TIME-SCALE Modifications

     . TM=0.3 (shrinking):    1    2    3        . TM=1.3 (stretching):    1    2    3

CONTINUOUS-like contacts variations

independent time-scale (TM) and granularity (Gran) modifications
       1 :   TM=1 - Gran=0.45        2 :   TM=1 - Gran=4              3 :   TM=0.3 - Gran=1        
       1 :   TM=0.4 - Gran=2          2 :   TM=1.2 - Gran=0.08      3 :    TM=1.4 - Gran=0.4

Audio Content Variations

only the rhythmic pattern of the original recording is kept
    Breaking Glass 1                       Breaking Glass 2          Breaking Stone/Ceramic 1
    Breaking Stone/Ceramic 2        Breaking Stone            Sliding Wood

Results of Retargetting to an Interactive Physics-Driven Animation

(game-like setting)

          A Stone                   A Wood Cylinder              A Ceramic Duck