• Welcome to the all-new HUG. All content has been converted from the old system, and over the next days we will re-style HUG in a more familiar way.

The proper evaluation of audio equipment - where do we go from here?

A.S.

Administrator
Staff member
Pantone colours and grading sound

Pantone colours and grading sound

... Your point about after longer change over periods would surely make one more uncertain ( opposite to what the essay I attached suggested)....
Thanks for the appreciation. In fact, I made the clips late last night (after returning from the BBC) and playing them this morning, I didn't think they were adequate to convey the point. So I spent a couple of hours experimenting and re-making etc..

The problem I have here is that I take all of this for granted - tools of my trade - that its easy to forget just how revolutionary the A-B concept is to outsiders. My neighbour is a fashion clothes designer and has a house full of colour charts so that when she instruct the factory (in Pakistan, China or Vietnam - sometimes the same garment made simultaneously at all three for supply security) she defines exactly what colour numbers to use. The factories have the same printed books of Pantone (or similar) colours and side by side, all three garments should look identical when compared on her desk in London, under neutral light and against her colour chart books which are very expensive as the confounding variable in the printing process have been tightly controlled. Those are the tools of her trade, and to suggest that she burns her Pantone books and relies on her colour memory would bring on one of here famous quizzical looks and the question 'why the hell would I want to work blind?' I'm sure.

The point is that in the world of colour manufacturing there are absolutes, and those absolutes are communicable to others using the standardised charts and tables you can see here. The essence of controlling colour is making visual side-by-side comparisons, and even better than that where matching A with B is super-critical (and get it wrong and 'see you in court') you use an instrument such as here and here and here. All we are doing with a sonic A-B is using our ears as that sonic A-B Pantone chart!

Can you just clarify what you mean in the above quote? I want to be sure that we are absolutely in-step. Thanks.
 

P.C.

Member
Clarification

Clarification

Sorry the quote was rather clumsy (rushing to get to work!) I was just agreeing with you that trying to compare the separate clips 1 and 1 b after any period of time ( tea making, cable swapping etc) would be very hard if not impossible.

Where as the author of essay seemed to suggest that this was the correct method to evaluate differences.
 

A.S.

Administrator
Staff member
Increasing the inter-gap between event A and B

Increasing the inter-gap between event A and B

Sorry the quote was rather clumsy (rushing to get to work!) I was just agreeing with you that trying to compare the separate clips 1 and 1 b after any period of time ( tea making, cable swapping etc) would be very hard if not impossible.

Where as the author of essay seemed to suggest that this was the correct method to evaluate differences.
OK, good.

You prompted me to ask myself what would the consequence of 'slipping' the delay between A and B from the mechanical time-of-flight of the relay contact leaf-spring (about 0.015 sec) to an six times longer - 0.1 Second. So I made another example (different switch-points, not directly comparable with Clip 2 - forgot to save the master file with cue points). Would you say that a gap of one tenth of a second made comparison of A v. B easier or not? Or just a slightly different, but not necessarily better or worse way of comparing?

Clip 3 - longer gaps

I have my own view on this but both Clip 2 and 3 are unrealistic for two reasons: you the listener have no control over the switching as I've done that for you. If you did have control (and maybe this is where I unwittingly differ in methodology from some of the prior art) you would expect to hear a difference and would mentally prepare yourself for that and to make sense of it. After all, you are not going to the trouble of setting up your own A-B test to hear that A and B sound the same: you are anticipating that they won't and are tasked with analysing that difference as part of your project workflow. It's not a leisure activity for fun! Secondly, we are just using a pre-recorded audio file. The practical use of this is to compare loudspeakers which fire sound into the room over 180 degrees+, some of which returns to our ear colored by room reflections. Depending on the dispersion of the speakers (etc. etc. etc.) what you will hear will be a little or a lot different between A and B.

So what I will try and record over the next days is what a dummy head stereo mic at the hot spot actually hears when comparing two speakers using the A-B relays. I'd expect that to me a much more interesting test.
 

P.C.

Member
Distracting pauses?

Distracting pauses?

Well I found the delayed switching much more of a distraction. It made it harder to focus on the continual musical line.I think there was only six switch changes in clip 3 and of those I could discern a change in tone on about 4 of them. On the original clip 2 I heard 12 of the 13 switch overs and of these probably heard 10 changes to the tone.
The clarity of those change did depend somewhat on what instruments and notes happened to being playing at the time of each switch .Which does suggest that the choice of music/ instrumentation may play a part in how easily or not a difference can be heard.

This was my earlier concern as regards the continual changing nature of music. But as you said an audio loop along with shorter or longer gaps selected between the switching would hopefully overcome those problems . I should add that i listened to the clips through headphones. It will be interesting to hear what the dummy head stereo mic ( binaural?) adds to the test.
 

EricW

Active member
Self-deception

Self-deception

It's an awkward situation when someone who is so desperately hunting for a compliment indicates what they want you to re-affirm; you are torn between telling an untruth which merely perpetuates their delusion or steering the subject away leaving the question hanging in the air when you both know it is unresolved. What is kinder? What is cruel? Can her self-image and the reality ever be re-aligned? Should I try to play along even though my brain just would not allow my mouth to utter those affirmative words because they were just not true (even though it is ungallant not to)? Is it my duty to do anything other than drink and listen? Do I care? Yes I do; I'm reliving the three hour conversation ...
I don't know if you own a Kindle but if you do I can recommend a rather fascinating essay called "Lying" by the American writer Sam Harris (as far as a I know it's only available as an e-book on Amazon). The case he makes is that even the little social "white lies" that we nearly all tell - and you neatly describe one way in which the pressure to do that arises - are morally questionable and ultimately perhaps as harmful as what we think of as "real" lies, because we reinforce people's illusions rather than helping them see reality. It's a bit more complicated than that, of course, but that's the general gist of it.

And Zemlya, yes, of course, I believe historians are capable of rational thought! Sorry if it seemed that I suggested otherwise - it's certainly not what I meant. I don't think you have to be a scientist to think rationally (that would certainly exclude me) or use elements of the scientific method: you just have to keep an open mind and try to be persuaded by evidence rather than belief. Sounds simple, but it's apparently a challenge for many (and yes, a little knowledge doesn't hurt either, as long as you don't use it to erect belief systems that go way beyond the evidence).

I have done some quick switchovers between audio equipment (back in the say when retailers had comparators that allow for that) but not the kind of A/B testing described on the HUG. However, it seems glaringly obvious to me that, if a reliable and knowledgable person adverts to scientific studies in which amplifiers have been determined to be indistinguishable under A/B conditions if levels are properly controlled, then the onus is one anyone holding a different view to (1) demonstrate how/why the existing studies are faulty, and (2) demonstrate how differences are audible under blind, level-matched conditions. How can one possibly be seen as credible if one refuses to do that?

Even if you grant some credence to the idea that prolonged (solo A or B) listening will reveal subtleties and nuances that A/B testing won't, and accept that it's not just all psychological artefacts (which prolonged listening probably is), shouldn't you at least be prepared to do an A/B test after you've acclimatized to the mysterious differences and have sensitized yourself to what they sound like? I mean, once you know, you know, right? If not, why not?
 

A.S.

Administrator
Staff member
Well I found the delayed switching much more of a distraction. It made it harder to focus on the continual musical line.I think there was only six switch changes in clip 3 and of those I could discern a change in tone on about 4 of them. On the original clip 2 I heard 12 of the 13 switch overs and of these probably heard 10 changes to the tone. The clarity of those change did depend somewhat on what instruments and notes happened to being playing at the time of each switch .Which does suggest that the choice of music/ instrumentation may play a part in how easily or not a difference can be heard.

This was my earlier concern as regards the continual changing nature of music. But as you said an audio loop along with shorter or longer gaps selected between the switching would hopefully overcome those problems . I should add that i listened to the through headphones. It will be interesting to hear what the dummy head stereo mic ( binaural?) adds to the test.
I agree. Getting the best outcome from the A-B comparison seems to benefit from a short interruption, and yes, I absolutely agree that the sonic comparison is heightened or somewhat suppressed depending upon what the music is doing at the point of switch-over. If speaker B has, for example, a boosted HF response compared with A, and if by coincidence a cymbal is hit just at or on the point of change-over then B will be distinctly obvious. But if a cello is playing at change-over then, assuming the loudness of A and B are similar, there will be no detectable sonic difference.

However, this is not a major inconvenience. The listener has full control over the switch operation and will be aware that he needs to randomise the operation (especially in the all-critical early minutes where he is bombarded with sonic data) and to switch backwards and forwards when curious about how an instrument sounds on both. Secondly, he has complete control over the musical library. It's no bad thing that he takes mini-breaks to get up and change a CD or he can pre-burn a CD with a musical selection and just plough through it. Or both. Or any other strategy that suits him. All that matters is his opinion at the point of comparison which, over the minutes after many comparisons, will settle down to a long-term view which really doesn't change.

If he does take a break, then don't make a mental note of whether the switch is on A or B - just leave it on whichever one sounds "best" without trying to double guess what it is, take a break and return to playing the music without touching the switch-over control. Let it play for a minute or so and change-over. Does the pre-break "best" now still sound "best" after the break? Often it doesn't because the pre-break has seduced the listener with a particular sound. I've been round this loop so many times that I know exactly how you (want to) fall head over heels in love with your new creation that you've spent months on. But if you don't get a grip on the reality that in truth, it doesn't sound as good as an older model, you will never bring to market a truly superior design. Your income, your pension, your lifestyle depends upon total self-honesty, no matter how painful that may be. If you can be honest early in the design you stand a good chance of minimising wasted effort.
 

EricW

Active member
Credible article

Credible article

http://people.xiph.org/~xiphmont/demo/neil-young.html#toc_lt

Not sure if the above link will work...but it may be of interest to read, to get another confirmation of the validity of ABX testing. And of level matching:)
What a great article, Kumar. Thanks.

I particularly liked this paragraphs, as it seemed to applicable to much of the recent debate on HUG, and is (I venture to say) a truth of general application, with validity beyond audio:

How to [inadvertently] screw up a listening comparison

The number one comment I heard from believers in super high rate audio was [paraphrasing]: "I've listened to high rate audio myself and the improvement is obvious. Are you seriously telling me not to trust my own ears?"

Of course you can trust your ears. It's brains that are gullible. I don't mean that flippantly; as human beings, we're all wired that way.​
 

Kumar Kane

New member
Human ears

Human ears

What a great article, Kumar. Thanks.

I particularly liked this paragraphs, as it seemed to applicable to much of the recent debate on HUG, and is (I venture to say) a truth of general application, with validity beyond audio:

How to [inadvertently] screw up a listening comparison

The number one comment I heard from believers in super high rate audio was [paraphrasing]: "I've listened to high rate audio myself and the improvement is obvious. Are you seriously telling me not to trust my own ears?"

Of course you can trust your ears. It's brains that are gullible. I don't mean that flippantly; as human beings, we're all wired that way.​
Eric, you are welcome. I thought that the part about hirez audio, another controversy like the amp one, might be of interest to people here too.

Of course we are all wired that way, ears are just transducers, it is the brain that hears, usually what it wants to, influenced by lots of things beyond what the ears pass on.

I also liked the comments about louder music sounding better in quality -and that "stereo salesmen have known this trick for a long time".
 

DSRANCE

Member
Art plus specs ...

Art plus specs ...

Please may I return to the basics for a moment?

When I was in the industry in the early days of the mid 70's, I started with every speaker being chosen by a relay switched comparator system. For initial basic comparisons, it was a boon and later on, when "single speaker (pair) dems" became the norm in the UK, we used a manually selected hard-wired comparator for initial parts of the dem and the choices of two or possibly three models were then demonstrated over a longer time, in hopefully relaxed circumstances away from the hustle and bustle of the main shop. Worked well for us and we had very few returns I remember.

Coming more up to date, and please forgive me repeating myself, but the memory is so clear and imprinted, quick initial listening between the Rogers LS5/9, M30 and M30.1 didn't really show much difference to me, especially in the warmer Summer months where I suspect the Rogers drivers had "loosened up" a little. More extended listening though, showed a greater sense of "ease" in the WAY the music touched my soul. I found the M30.1 easier to "suspend disbelief" if you see what I mean.

I hope the comments above make some sense and don't cloud the issue being discussed in such depth. of course "HiFi" gear cannot take the place of a live performance, but the best examples allow one to forget this for a while and for me, it's where basic measured specs leave off and the art takes over.
 

A.S.

Administrator
Staff member
BBC LS5/9 (1980s design)

BBC LS5/9 (1980s design)

... Coming more up to date, and please forgive me repeating myself, but the memory is so clear and imprinted, quick initial listening between the Rogers LS5/9, M30 and M30.1 didn't really show much difference to me, especially in the warmer Summer months where I suspect the Rogers drivers had "loosened up" a little. More extended listening though, showed a greater sense of "ease" in the WAY the music touched my soul. I found the M30.1 easier to "suspend disbelief" if you see what I mean...
I wonder if you'd confirm whether you auditioned these speakers at the same time/place more or less side by side or if you are conveying an impression separated over a time period. I ask that in particular of the LS5/9 (which you've mentioned a few times elsewhere) - do you own 5/9s? We have many pairs of these traded in by the BBC over the years (for Monitor 30s).
 

DSRANCE

Member
Suspension of disbelief with M30.1

Suspension of disbelief with M30.1

I wonder if you'd confirm whether you auditioned these speakers at the same time/place more or less side by side or if you are conveying an impression separated over a time period. I ask that in particular of the LS5/9 (which you've mentioned a few times elsewhere) - do you own 5/9s? We have many pairs of these traded in by the BBC over the years (for Monitor 30s).
I have been given the custody of a pair of late? non BBC *** Rogers LS5/9's which were used in a video editing suite for around a year now. Since I haven't paid for them, I cannot sell them on and am thrilled that they sound as good as they do, at least in Summer months. The comparisons I've referred to were done a few times at the same venue (hifi dave's place) and over a half hour period each time. I readily admit that this comparison was totally informal, so please accept it as such. The differences for me at any rate were of a "ease of suspension of disbelief" nature, the M30.1's making the music easier to become involved with over a few minutes listening, this continuing for me with a broad smile on my face :)

Forgive the non-objective paragraph above, but for me, the emotional content of music reproduction is of vital importance. Since I think we're all agreed that reproduced music is a varying facsimile of the original recording or mixing session, I feel that getting myself involved in the reproduced performance is as important as the tradidional loudspeaker aspects such as neutral response and low colouration/distortions/dispersion.

I can't remember where I read this comment, but I read with affectionate amusement recently that Quad's Peter Walker never really listened TO music through the amplifiers he so successfully designed when developing them. Rather, he listened for faults/distortions and jiggled the compromises to minimise any audible effects these may have had on the reproduced sound. This is well evident in the utterly charming model II valve amps of yore - hopelessly coloured, but so lovely to listen to and through. I suspect that this couldn't be done so easily now, since modern designs have come on so much further that differences between them are possibly more to do with clipping performance and damping factor as anything else.

Sincere apologies for any thread drift. I hope the above is fairly relevant in any case...


*** These LS5/9's still retain the fine tweeter level adjustment via wire links on the crossover board, but lack the tappings on the front baffle that the BBC models had I believe. Since both examples sound the same and have excellent stereo imagery, I'm rather hoping that they haven't "gone off" too severely, and I feel this has been borne out in the M30 comparisons.....
 

A.S.

Administrator
Staff member
Rounding-up on audiophilia

Rounding-up on audiophilia

I recently attempted to start a thread on the realities of amplifiers in the main forum but I don't have the time to manage it. The thread content still exists; it will have to wait for a new home.

I am certain that under tightly controlled A-B conditions all the amplifiers I have examined are instinguishable. It's not a matter of of listeners having a preference, they simply cannot be told apart. My list is not exhaustive, and clearly the amplifiers must meet a certain minimum performance threshold or they would be identifiable. If, for example, due to poor design or component layout, one amp had a buzz or whistle, it would be surprising if it were not identifiable in A-B conditions. I am so sure of this that metaphorically I would 'bet the company' on this opinion. Although it is many years since I have conducted similar comparisons of speaker cables, I can see no reason to reach a different conclusion.

If these findings remain true - and they are not new results; every A-B comparison I'm aware of conducted by others has reached exactly the same conclusion - then the audiophile has a whole industry of bullshit moulding his perceptions. Since we know that marketing people and politicians are expert at manipulating public perceptions (sometimes with disastrous consequences as we've seen in history) and we also know that consumers are, frequently, fantastically gullible, there really is no rational way of breaking this cycle. I should not use Harbeth's platform to try and so so, nor should I devote the ridiculous number of hours I do to this enlightenment project. One or two rational people like me are easily swamped. Whilst I can appreciate that with technical equipment like home audio, there will be few who even remotely understand the details under the hood, I am constantly astonished by the total - yes, total - failure to understand how unreliable the ear is as a reference tool despite its capabilities being thoroughly researched and well documented and understood. To place all ones faith in the ear is a denial of reality.

I've decided that a far better use of my time is to lay down, incrementally and outside HUG in a separate blog and without interruptions and distractions, how ill advised the audiophile is to trust his ears. I care not if I convert anyone - my sole motivation is to get this down on record so that, seen from a historical perspective, I am appraised as a rationalist in a sea of the deluded. This goal is very important to me. As this project is going to take months/years at my own pace (and may lead to a printed publication at the end, so has to be carefully structured with appropriate cross-referrals) I intend have no choice but to reduce my huge commitment to HUG to make time for this important project.

I am exasperated, frustrated and saddened by the mental disorder known as 'audiophilia'. It is a miserable state of being with no remission and has little or nothing to do with music. It is highly addictive. No good has ever or will ever come from it. My blog space is being created and its existence will be made know as and when. I am not encouraging debate about issues that I have studied for over thirty years and are supported by fact. In a democracy other views are encouraged to appear in other forums - not mine.

Thanks for your understanding.
 

SDRichard

New member
What about .... vitamins!

What about .... vitamins!

If these findings remain true - and they are not new results; every A-B comparison I'm aware of conducted by others has reached exactly the same conclusion - then the audiophile has a whole industry of bullshit moulding his perceptions.
If it makes you feel better, Alan, this is not at all unique to the audiophile world. It is well documented that placebos work very well in medicine. What about the vitamin industry? Many clinical trials have shown that vitamins and other food supplements do nothing for a person maintaining a normal diet, yet it is one of the fastest growing industries.

I admire your engineering prowess, integrity and brutal honesty. But being a wise man, I'm not so sure. :)
 

A.S.

Administrator
Staff member
Just back-up the chit chat with facts ....

Just back-up the chit chat with facts ....

If it makes you feel better, Alan, this is not at all unique to the audiophile world. What about the vitamin industry? :)
I hadn't thought of that! The difference is - I suppose - that people quietly get on with taking their vitamins or whatever; they don't feel the urge to tell everyone who will listen how their life has been transformed by them. Buy when it comes to audio electronics and accessories, we've all got to be told, whether we believe, care or are in the slightest interested. What goes on behind closed doors is fine: I (and other rationalists) just don't want to hear about it!

This really is my only irritation. I certainly want folks to go to their audio dealers and spend $$$$ - as much as possible and more - on electronics etc.. That's good business and happy dealers. What I don't want to read any more of on HUG is one consumer romanticising his purchase and by endorsement here, teasing another into action. The difference in electronics are very, very, very, very small. That must be obvious to anyone who knows how they actually work and how little scope the designer has for doing anything different to his competitors whilst maintaining the all-essential reliability and durability. That closes off all the design options - completely.

Sure, you can take an ordinary car engine and fuel it with nitrous oxide. It will hugely enhance the performance. But not for very long. Is it absolutely understood that electronic designers are hemmed-in by many constraints? They do not have a blank sheet of paper to express their desires. It's 90% filled-in with constraints before they even start.

Oh there is one more thing. For about $30 and a couple of hours bench assembly time (a one off investment) and a borrowed test meter (cost perhaps $150) anyone - absolutely anyone - could make their own A-B switchover box. Armed with that they could put to rest once and for all whether audio item A is truly superior (or even distinguishable) from item B. Is it really too much to expect such a piffling personal commitment to a hobby that absorbs thousands of hours and dollars? Any posts which following the use of the comparator reports 'I carefully calibrated the loudness between item A and B, removed as many variables as I could but upon listening could positively distinguish A from B on 80 out of 100 switch-overs' would be taken very seriously indeed here. But if contributors don't or won't make what is, against the importance of the outcome, a rather tiny personal effort, why should we here give the oxygen of publicity to chit chat which is nothing more than emotional highly personal subjective preferences? We shouldn't and we won't be any more.

The tools to turn subjective meanderings into tightly focused objective comparisons are cheap, as complex as wiring a battery operated electric door bell and fool proof. What earthly objection to using them is there?
 
Top