A/B Testing: 5 Tips for Testing Audio Gear

0

The gold standard in determining the sound quality of a microphone, mic preamp, recording interface or another piece of equipment is through the use of A/B comparisons. Even the most expensive equipment has limitations, and it’s best to evaluate equipment relative to similar gear. In its essence, A/B testing means putting two pieces of audio equipment side-by-side in identical conditions, then comparing the results. The process is quite simple and surprisingly underutilized for gear comparisons.

The most important factor in A/B comparisons is keeping conditions as close to identical as possible. Both acoustics and psychoacoustics play a major role in how we perceive sound and shouldn’t be underestimated. For example, the same recording played only one decibel louder will generally sound “better” to the human ear. In addition, we’re incredibly biased to perceive more expensive equipment as sounding better than less expensive equipment. Armed with the proper information, a less-than-perfect test can still yield usable results as long as the conclusions take into account the various factors that might bias outcomes. Here are five tips for conducting proper A/B gear comparisons:

Research Gear Specifications

The most important step is to understand the technical differences between two pieces of gear and to look for obvious differences that might noticeably affect the results. For example, a unidirectional (e.g., cardioid) microphone will behave very differently from an omnidirectional microphone. An omnidirectional microphone will pick up sound from all directions relatively evenly and lack the proximity effect of directional microphones. In other words, omnidirectional microphones don’t vary in bass response as the sound source varies from the distance to the microphone. Basic differences in electrical circuitry can make a difference as well. Microphones and preamps with output transformers can provide small amounts of compression that result in minute increases in loudness. Compression can be added in software quite easily in place of an output transformer, meaning that transformerless equipment isn’t necessarily inferior. Obviously, this research can be taken to the extreme, so we recommend taking two or three minutes to research manufacturer specifications before conducting any A/B gear comparisons.

Matching Sound Sources

Though it may seem obvious, it’s fairly common to see equipment comparisons using different recording takes, separated by time. Everything from the distance to the sound source, loudness, change in acoustics, and quality of performance can all make a difference in final results. If comparing equipment in this manner, consider the results preliminary, and be wary of drawing firm conclusions without further testing. If comparing microphones, for example, we recommend using a loudspeaker in a fixed location, playing only one audio file for both tests. When comparing microphone preamps, DI boxes, or audio interfaces, we highly recommend using identical recorded sound sources for each test.

Matching Loudness

This is the biggest challenge in A/B comparisons, as loudness makes the largest difference in whether or not one recording sounds better than another. Loudness can never be perfectly matched due to mismatches in electrical circuitry, so use a combination of your ears and metering to match levels meticulously before doing any side-by-side comparisons. Keep in mind which of the two sound louder in your results, and retest at multiple loudness levels if possible.

Compare Blind

It’s impossible to conduct a self-created double-blind test, but make it as difficult as possible to know which of the recordings you’re listening to when doing A/B comparisons. Once you’ve matched the levels between both sources, export short files and load them into a music player app like iTunes. Shuffle between the two, and consider adding a third similar, “dummy” recording into the rotation. Listen with your eyes closed and observe the differences. Pick your favorite of the recordings over multiple attempts, and try to predict which sound source is which with greater than 50% accuracy.  More often than not, the differences will hardly be noticeable.

Experiment with Equalization

Occasionally, the only substantial differences between equipment are minor differences in frequency response. With some minor EQ, it might be possible to make a $300 microphone sound nearly identical $3000 microphone. Try to match the sound of an inexpensive microphone to a more expensive one using EQ and a frequency spectrum analyzer. If it works, save the EQ curve as a preset and use it in the cheaper microphone in place of the more expensive one.

Summary:  

You shouldn’t need a degree in electrical engineering to compare two pieces of equipment side-by-side. Feel free to take shortcuts, but keep in mind the ways in which our minds are inclined toward bias, especially with regard to equipment price and loudness. When watching gear reviews online, look for errors in testing methods, and keep in mind the types of bias that skew results. For further study, we highly recommend watching Ethan Winer’s “AES – Damn Lies” workshop on YouTube, referenced on our Audio Resources blog post. 

Have you ever done any A/B gear comparisons? If so, what were your results? Feel free to let us know in the comments.

Photo credit: Eryk Thompson

 

CEO and Founder of Finewav. Instagram: @ErykThompson Vero: @ErykThompson

Leave a Reply

My New Stories