Reverse Conducting Plots and Analyses
Craig Stuart Sapp <craig@ccrma.stanford.edu>
25 Sept 2005 -- 8 October 2005

Loading the raw data for analysis

Load the reverse conducting durations for each beat from the preprocessed data file for Mazurka Op. 7, No. 3.  The data is stored in an array which contains the duration between beats in milliseconds.  Each row in the array represents a separate reverse conducting trial, and each element in the trial is duration to the nth beat of the trial.

Here are the durations in millseconds between the first and seconds beats (measure 1, beats 1 and 2) of Chopin's Mazurka Op. 7, No. 3 performed by  Ignaz Friedman in 1930 for each of the twenty trials:

Looking at basic statistics for a beat

Finding the minimum and maximum duration of the first beat:

The average value for the duration of the first beat:

Find the average beat duration for each beat in all trials:

Now, look at the timing spread between the average beat and the actual beats to determine the basic reproducibility accuracy of the reverse conductor.

Here are the differences between the first average beat duration and each individual trial's first beat (in milliseconds):

Display the distribution of all beats from all trials with respect to the average beat position of each beat:

<< Graphics`Graphics`

This data has a nicely shaped distribution, so  meausre some basic statistics about it:

The standard deviation of the beat locations are not necessarily constant for each beat, but it is reasonable to assume that the beat accuracy is constant according to the picture above where 95% of all beats between different trials fall within +/- 60 milliseconds of the average beat positions for each trial.  This means that the reverse conductor in this can reliable repeat his beats for this particular performance within 60 millseconds of his average beat location 95% of the time.

Now calculate the confidence bounds based on the Student distribution with a confidence level , by first loading the Mathematica package which calculates confidence intervals:

This confidence interval signifies that the actual mean beat duration of the first beat has a 95% probability of being between 363 and 412 milliseconds.

Plotting duration curves for the composition

Here is a plot of the average durations for each beat over the 20 trials of reverse conducting on the composition.

Now add the 95% confidence ranges to the plots as well as the maximum and minimum range for the duration of each beat.  The confidence levels are displayed in dark gray, while the min and max range is displayed in light gray.

First, zoom in on the first 20 beats so that detail in the plot can be seen:

Now, below is shown the beat-duration curve for the entire piece.  Notice that the tempo of the opening 50 beats (The A section)  is repeated in the return of the A section starting at beat 230.

Displaying duration (tempo) changes in the performance

As a first approximation, the expressivness of the performance can be interpreted as changes in the tempo.  The greater the tempo change, the greater the expressiveness.  So, let's display the changes of the duration plots given in the previous section.  The following plots display the acceleration and ritard of the performance on a beat by beat level.  

Now, Average the data with a moving median of 10 beats to smooth out the previous plot.

<< Statistics`DataSmoothing`

Phrase structure is visible in the following plot of the smoothed changes in duration.  Each arch in the plot below represents a phrase (although some are elided towards the end).  In particular note that the A section phrases are clearly marked, while the return of the A section starting at beat 255 shows a blurring of the phrases.

In the above plot, the phrase starting around beat 125 to beat 150 (bar 41 to 50) is flatest phrase which means that the tempo remained fairly constant throughout that region of the music.  This portion of the music is the beginning of the B section of the piece where the music becomes more fanfare-like and military with wide ranges in dynamic.  The most "expressive"phrase starts at beat 205 (measures 68 -76 ).  This is the point where there is a second repeat of a melody in the left hand, and marks the recapitulation to the A section in measure 77.

Here is an interpretive segmentation of the music.  Red lines indicate phrase boundaries identified by eye; the green lines indicate "missing" phrase boundaries.  The first missing phrase boundary is at the peak around beat 215, and the seoncd missing phrase boundary is in the valley at beat 290 where there may be a intentional phrase boundary, but it is very shallow and broad.  It is interesting to note that the opening phrase of the piece and its return at beat 250 are similar, but the music of the 2nd and third phrases which is repeated starting at beat 270 have very different expressive tempo qualities.

Displaying uncertainty in reverse conducting

Here is a plot which shows how difficult it was for the reverse conductor to match the beat of the performer at each beat in the composition.  The red line is a plot of the difficulty of following the performer.  The black line in the second plot below shows the tempo (beat duration) fluctuation plot from above for comparison.  Peaks in the black plot line up well with the peaks in the red plot.

Not much interpreting of this type of data yet.  Basically the more the tempo changes between beats, the more inaccurate the reverse conductor is at locating the next beat properly, which is to be expected.  It would be interesting to determine if certain categories of tempo change cause uncertainty in the reverse conducting while other categories of tempo change do not cause uncertainty.  For example, offbeats can be used by the performer to indicate the rate of a tempo change before the new beat actually occurs.

Displaying data as tempo per measure rather than as duration per beat

Musicians are more familiar with tempo, so the following plots convert duration in milliseconds into tempo in beats per minute.  Also, since this composition is fast, and the tempo is indicated as 54 measures per minute (dotted half note tempo) in the 1915 Schimer edition of the mazurkas edited by Rafael Joseffy, the tempo values are divided by three (three beats per measure) to compare to this metronome marking.

Now display the min, max, and confidence intervals on the tempo plot for the entire composition:

Numbers on x-axis in plot below indicate the measure numbers.

Zooming in on the tempo curve for every 8 bars

The previous plot showing the tempo curve for the entire piece is difficult to see detail, so now break up the tempo plots into 8 measure segments.

Plotting variations: adding dots at beat points

Plotting variation: Squared plot

Plotting Variations: Displaying all trials points in a tempo plot

Size and color indicate the individual trials of reverse conducting so that you can distinguish individual tempo curves.  Earlier trials are large red circles, later trials become smaller circles which progress through the colors of the rainbow.

Plotting Variations: Aligning and plotting absolute time of beats

The above plot shows the absolute duraton location of each beat for each trial.  The bottom axes labels are in seconds, the top axes labels are in measures, and the right side axes numbers are the trial numbers, starting with the first trial at the bottom of the plot.  This plot assumes that all trials started beating at the same point in the recording, which is not physically possible, so the next section tries to identify the offset between each trial by minimizing the difference between beats in each trial compared to the first trial.

Aligning the absolute-value beat times between trials

First, do a test alignment between the first and 18th trials.

Identify the lowest point in the parabola which indicates the relative shift between performances.

Therefore, the 18th trial starts beating the first beat 8 milliseconds before the first trial does.  Now, calculate the offsets for all other trials against the first trial.

The average shift is 20 milliseconds:

Compare to the non-offset trial data plot which is repeated below, and notice that the points representing conducted beat locations fit to the average beat positions better in the above plot:

Offset adjustments to timing values to align with audio recording

Up to now the timing data for the human beats has been referenced to the first beat in the score.  This beat was initially assigned 0 milliseconds.  In the section above, a relative shifting of each trial was done against the first trial using a least-squares fit to coordinate the individual trials in absolute time.  

In the actual recording, the music does not start at 0 milliseconds into the file, but rather starts after an initial segment of silence.  In theory, the duration of this silence is easy to measure from the soundfile; however, the actual 0 point in the human beats does not necessarily coincide with the beginning of the first note, and it would be expected to be about 50 to 250 milliseconds later due to human reaction time.

Therefore, a more accurate method of determining the initial offset of the mean human beat timings into the soundfile is to use a least-squares fit to align the human beats with beats measured accurately in the audio file.  First, load the beat data measured directly from the soundfile into a graphical display from an audio editing program:

This data contains the most accurate measurement of the beat positions from audio events in the soundfile, probably within an accuracy of 1 millisecond, but definitely within an accuracy of 3 milliseconds which is better than the accuracy of 5 milliseconds of the input tapping method using the computer keyboard in Windows XP.  

The events were chosen due to their visual clarity (mostly of the amplitude envelope), caused by accented beats, or beats starting in silence where the amplitude change is clear.  The first colum in the table below is the measure number, second column is the beat in the measure, and the last column is the time in milliseconds from the start of the audio recording (which includes the initial silence at the start of the soundfile).

Now, calculate the beat location from the start of the piece (where beat one of measure one is absolute beat 0).

The first beat is not accurately performed by the human, so remove it from the calculations.

Extract the same beat positions from the mean beat sequence:

Seq1 and seq2 below are the two timing sequences for carefully measured beat locations in the audio dat.  The first sequence are from the human beat timings which are reference from the first beat.  The second sequence are from the audio and are referenced to the start of the soundfile which contains some initial silence.

Here are the time difference between the human beats and the audio beats which range between 2350 and 2550 milliseconds.  The average if this difference will be used to shift the human beats to their positions in the audio file.

Now, let's calculate the needed shift (2460.61 ms) the more difficult way just to check that this is the best shift to minimze the offset between the audio and human beats.

On the average, the measured beat positions is 40 milliseconds away from the measured position in the audio file.  The worst case for the beat difference for the measured audio beats is 127 milliseconds.

Output data

Store the data analysis information into a text file for future reference:

The full analysis output file contains an information entry for each beat (except the last beat in the input) with seven numbers on a line:
   (1) the expected position of the beat in the audio file in milliseconds
   (2) the average beat duration for all trials
   (3) the minimum beat duration in all trials
   (4) the maximum beat duration in all trials
   (5) the minimum location of the true average beat location with 95% confidence
   (6) the maximum location of the true average beat location with 95% confidence
   (7) the standard deviation of beat durations for all trials
   
   Here is the information for the first beat in the Mazurka: