This is a very good point that you and busb make. ABX testing by definition has one very big limitation - it can only prove a positive result but it can't prove a negative one.
Basically if you really can hear a difference then you will get a positive result which categorically proves that you heard a difference. That's all fine and dandy and provides good quality useful data. However the problem with ABX testing is that you can't prove a negative result. If you can't hear any difference it doesn't prove that a difference can't be heard. It just shows that those people on that particular occasion didn't hear a difference. If they tried it again another time they might be able to hear a difference.
I fully agree with you that ABX testing isn't perfect. But it's still the most reliable comparison method available for checking whether or not you can hear any differences between two things. Don't forget that it's not just ABX which can't prove a negative result but all other listening comparison methods can't either.
I'd say ABX as a methodology is so flawed - it's almost pointless. Some say it's the best we got - maybe but I say the best we've got ain't good enough or fit for purpose! In other words it's only going to convince those that have made up their minds that no differences exist. They are not the ones that need convincing!
Any method must be able to prove a negative as well as it can prove a positive in equal measure! There are ways to improve the reliability of ABX though. Scattered throughout the test sequences, their needs to be checks that all but the deaf will hear. These checks could be added harmonic distortion or channel imbalance for example. Any subjects failing to notice these checks could be discounted. The problem is that it doesn't take very long before subjects get very confused as to whether they can hear differences or not - tests need to be short with plenty of breaks to keep subjects awake.
My suspicion is that many people have been so convinced that ABX testing is flawless that they discount their own hearing prowess a little too quickly.
"The optimist proclaims that we live in the best of all possible worlds - the pessimist fears this is true."
James Branch Cabell
MAIN: Apple TV2, Mac Mini (controlled from various iThings using Remote), CA Azur 751BD & Panasonic P42V20B into audiolab M-DAC, feeding a Primare A34.2 class D power amp via XLRs, 2x 5m of Atlas Ascent 2 firing up Totem Arros. DALI Kubik Free in my kitchen
ON THE HOOF: iPhone 5S/Sennheiser MM450.
Same with Expectation Bias (imo).
"Everything has been said before, but since nobody listens we have to keep going back and beginning all over again." André Gide
My point being they only seem to work when the differences are extremely obvious.
That's the whole point though - some people were claiming there are extremely obvious differences but another group of people disagreed and sought to challenge those claims.
People talk about night and day differences all the time which is at odds with all the test results. Actual test results are showing that any differences are very minor.
I have always thought that differences between stuff like amplifiers to be fairly subtle. It then becomes a personal choice whether or not the improvements are enought to justify changing something or not. IMO, the difference, albeit it subtle must reach a threshhold beyond doubt.
One can argue the efficacy of ABX or other blind tests, but the results of such tests always seem to contradict the claims of significant differences existing.
If obvious purported differences between equipment on test existed, they would show on any form of blind testing, but they never seem to.
In lieu of any other form of testing, meeting whatever criteria is deemed sufficient to prove some level of accuracy approaching conclusiveness, I'll go with what is currently available and appears to be 99% accurate.
One could always take the openminded standpoint until conclusive proof becomes available, but sometimes you need to act on the balance of probability and one persons opemindedness, is anothers naievity or gullibility.
Mac mini > AVI ADM9Ts
What I find sad, is when all this obsession with testing stops people just going out and LISTENING. Other than a minute percentage of "objective" audiophiles, (made up almost entirely of blokes), this is enough for most normal folk.
I've been accused of "scaremongering unnecessarily" over encouraging people to "try" cables; I find this ironic in the light of where people are being "told", that little of what they hear is real, unless of course, if it conforms to a certain version of "reality".
I've been accused of "scaremongering unnecessarily"" over encouraging people to "try" cables; I find this ironic in the light of where people are being "told", that litlle of what they hear is real, unless of course, if it conforms to a certain version of "reality".
I totally agree that people should listen to as much different HiFi equipment a possible to see what they like the sound of best. There is no substitute to actually listening with your own ears. This is just common sense.
But when it comes to things like replacement mains cables where in 99.9% of cases it will make no difference people should be aware of their own limitations when 'just listening'. In long term tests your auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play.
I agree that listening to equipment is of the upmost importance but with things like mains cables this is where methods such as instantaneous A/B switching come in useful during the selection process. When this is done the people who previously thought that they could hear subtle differences in long term sighted tests realise that when compared in instantaneous A/B switching tests there isn't really any difference in the sound between the mains cables. This has been proved many times and even you Cno admit to have never tried comparing cables in a blind test.
If anyone's interested why Cno was accused of scaremongering unnecessarily you can read it in the link below. Cno posted some incorrect information in post 15 on page 1 to which I replied.
PC > AVI Neutron Five 2.1
Sony NWZ-A847 64GB Walkman > Westone UM3x
Steve, I respect you enough not to fall out over this.
What I will say is, "Everyone" on here who has tried TQ Black, has heard a difference....that is so statistically significant, that it should not be dismissed so easily.
There are highly experienced members on here who are not gullable, and can hear the difference.....but they choose not to post, so as to avoid this exact situation. i would also suggest that the majority of people who give it a go, hear the difference........and this is despite the "nay brigade" telling them it's impossible.
My objection is not what you believe, or even that you tell people that it makes no difference...........but that you tell them not to try. I actually give people a little more credit than that. You are in effect telling them what is, and what is not worth listening to, which is tantamount to censorship....and all from a position where you have either tried very little, or not at all.
A cable dem by the likes of Chord or TCI might actually make you see things differently; but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth".
Now this is an ABX thread, so it's my last comment about cables.
...but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth".
And what happened to him?
Marantz M-CR603 • Rega R3 loudspeakers • AirPlay • Apple iPad Mini • Apple iPhone 5 • Apple iMac • Apple AirPort Extreme 802.11N • Humax HDR-Fox T2 • Panasonic TX-L32D25B • Sony BDP-S390
That's something which I like and respect about you Cno. We're able to have a difference of opinion and discuss things sensibly without being rude or getting upset with each other and we're still able to be friends and have a laugh and joke afterwards.
This is a perfect case example for this thread which is totally relevant here. I bet that 'everyone' who has tried TQ Black and heard a difference did so in a long term sighted comparison test. I'm not disregarding what these people say but under these comparison conditions your long term auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play too.
Now if they heard a difference in blind A/B comparisons or even passed an ABX test their claims would have far more credibility. We all know that long term sighted comparisons can be unreliable.
My objection is not what you believe, or even that you tell people that it makes no difference...........but that you tell them not to try. I actually give people a little more credit than that. You are in effect telling them what is, and what is not worth listening to, which is tantamount to censorship....and all from a position where you have either tried very little, or none at all.
I have no objection with people listening to things for themselves. In fact quite the opposite I urge them to go and listen to as much hifi equipment as possible.
While it's best to do blind A/B comparisons wherever possible I also understand that in real life situations this is sometimes too inconvient to do. This is why people should to be aware of their own ears/brains/hearing/memory limitations when comparing products in long term sighted comparisons which are notoriously unreliable.
Right back at yer Cno.
What I find sad, is when all this obsession with testing stops people just going out and LISTENING.
Not so. It tells you what you need to listen to.
By somebody else's definition it would seem.
Onkyo TX-NR818 / Tannoy DC4 speakers / Marantz UD-7007
AVI Lab Series CD player / various cables
I'm probably an objectivist rather than a subjectivist when it comes to believing what is the best way to go about selecting hifi equipment, if the sound produced is the sole criteria. I'm on the fence a little regarding AB/X though because I'm willing to accept that audio memory isn't fantastic, my own preferred choice would be just instantaneous switching between A and B without me knowing if B was just A being used again or if a switch had actually taken place, with the question just being can I hear any difference between A and B and if so then obviously personal preference enters the equation.
While I'm happy to accept that audio memory can let a person down I do find it interesting that it doesn't seem so unreliable during sighted comparisons, when swapping between speaker cables (for instance) I've found it fairly easy to hear and remember differences and yet, using a rudimentary form of blind A/B switching, when I'm unaware of which cable is in use I've found it impossible to detect the same difference - or indeed any difference. The other interesting finding was that after the blind comparison, returning to a sighted comparison the cables still sounded identical - the earlier sighted differences were no longer apparent.
I'm not sure the argument that 'I didn't expect to hear a difference but I did so it must be real' stands up, surely if a person didn't think there could be a difference they wouldn't have bothered making the comparison in the first place? When I first got some interconnects to compare to the ones I already had I thought there probably wouldn't be a difference and felt a little silly asking the dealer if I could try some at home, and yet why was I going to the trouble of trying them? Because a part of me obviously thought there might be something in it, even if it seemed silly. I'm also not convinced that experience is any sort of guard against any sort of subconcious bias, there are a number of videos on youtube about something called the Mcgurk effect (were what you see directly influences what you hear), one of them by a chap who has been researching the Mcgurk effect for over 20 years - he is still fooled by it every time and there is nothing he can do to change that.
So that's my thought on the debate and my personal experience backs up the fact that, for me, my subjective opinion isn't especially reliable. I can well understand why mags/websites/reviewers tend not to use level-matched blind testing though, 'it all sounds pretty much the same' isn't going to sell many mags is it? In fact, according to one very well regarded magazine editor, many audiophiles actively want there to be differences and the mag wouldn't sell if they didn't print them.
Synology NAS + ATV2 > ADM9RS
I think that is a respectful, fair and reasonable approach....well most of it anyway.
We are all inclined to be drawn to, and influeneced by people who make sense of the world, in a way that ties in with how we naturally think. It is genuinly hard to remain open minded, as it is all too easy to reject anything that doesn't fit conveniently with what we see as reality.......and I very much include myself in this.
As I age, I take a more tolerant, less "black and white" view of the world, and have found things are never quite as simple as I first thought.
Martyrdom isn't for everybody.
I'd say ABX as a methodology is so flawed - it's almost pointless.
So everything sounds the same? Really?
© 2014 Haymarket Publishing