270 posts / 0 new
Last post
busb's picture
Offline
Last seen: 1 week 5 days ago
Joined: 14/06/2011 - 20:08
Posts: 928
RE: Subjective/objective testing /AB / AB/X, thoughts.

steve_1979 wrote:

DocG wrote:
OK, to put busb's remark somewhat sharper (may I, busb?): what if the participants all suffered neurosensory deafness ànd were really honest? Not a single positive result would be reported, so there is no difference, right? Utter :hs:, of course! I'm aware this is a daft example; it's just to make the point clear. Smile

This is a very good point that you and busb make. Smile ABX testing by definition has one very big limitation - it can only prove a positive result but it can't prove a negative one.

 

Basically if you really can hear a difference then you will get a positive result which categorically proves that you heard a difference. That's all fine and dandy and provides good quality useful data. However the problem with ABX testing is that you can't prove a negative result. If you can't hear any difference it doesn't prove that a difference can't be heard. It just shows that those people on that particular occasion didn't hear a difference. If they tried it again another time they might be able to hear a difference.

 

I fully agree with you that ABX testing isn't perfect. But it's still the most reliable comparison method available for checking whether or not you can hear any differences between two things. Don't forget that it's not just ABX which can't prove a negative result but all other listening comparison methods can't either.

 

I'd say ABX as a methodology is so flawed - it's almost pointless. Some say it's the best we got - maybe but I say the best we've got ain't good enough or fit for purpose! In other words it's only going to convince those that have made up their minds that no differences exist. They are not the ones that need convincing!

Any method must be able to prove a negative as well as it can prove a positive in equal measure! There are ways to improve the reliability of ABX though. Scattered throughout the test sequences, their needs to be checks that all but the deaf will hear. These checks could be added harmonic distortion or channel imbalance for example. Any subjects failing to notice these checks could be discounted. The problem is that it doesn't take very long before subjects get very confused as to whether they can hear differences or not - tests need to be short with plenty of breaks to keep subjects awake.

My suspicion is that many people have been so convinced that ABX testing is flawless that they discount their own hearing prowess a little too quickly.

CnoEvil's picture
Offline
Last seen: 2 hours 42 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

busb wrote:

My suspicion is that many people have been so convinced that ABX testing is flawless that they discount their own hearing prowess a little too quickly.

Same with Expectation Bias (imo).

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

busb's picture
Offline
Last seen: 1 week 5 days ago
Joined: 14/06/2011 - 20:08
Posts: 928
RE: Subjective/objective testing /AB / AB/X, thoughts.

MakkaPakka wrote:

busb wrote:

My point being they only seem to work when the differences are extremely obvious.

 

That's the whole point though - some people were claiming there are extremely obvious differences but another group of people disagreed and sought to challenge those claims.

People talk about night and day differences all the time which is at odds with all the test results. Actual test results are showing that any differences are very minor.

 

I have always thought that differences between stuff like amplifiers to be fairly subtle. It then becomes a personal choice whether or not the improvements are enought to justify changing something or not. IMO, the difference, albeit it subtle must reach a threshhold beyond doubt.

Overdose's picture
Offline
Last seen: 2 hours 32 min ago
Joined: 08/02/2008 - 18:23
Posts: 3346
RE: Subjective/objective testing /AB / AB/X, thoughts.

One can argue the efficacy of ABX or other blind tests, but the results of such tests always seem to contradict the claims of significant differences existing.

If obvious purported differences between equipment on test existed, they would show on any form of blind testing, but they never seem to.

In lieu of any other form of testing, meeting whatever criteria is deemed sufficient to prove some level of accuracy approaching conclusiveness, I'll go with what is currently available and appears to be 99% accurate.

One could always take the openminded standpoint until conclusive proof becomes available, but sometimes you need to act on the balance of probability and one persons opemindedness, is anothers naievity or gullibility.

CnoEvil's picture
Offline
Last seen: 2 hours 42 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

What I find sad, is when all this obsession with testing stops people just going out and LISTENING. Other than a minute percentage of "objective" audiophiles, (made up almost entirely of blokes), this is enough for most normal folk.

I've been accused of "scaremongering unnecessarily" over encouraging people to "try" cables; I find this ironic in the light of where people are being "told", that little of what they hear is real, unless of course, if it conforms to a certain version of "reality".

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

steve_1979's picture
Offline
Last seen: 1 week 4 days ago
Joined: 14/07/2010 - 21:04
Posts: 3791
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

What I find sad, is when all this obsession with testing stops people just going out and LISTENING. Other than a minute percentage of "objective" audiophiles, (made up almost entirely of blokes), this is enough for most normal folk.

I've been accused of "scaremongering unnecessarily"" over encouraging people to "try" cables; I find this ironic in the light of where people are being "told", that litlle of what they hear is real, unless of course, if it conforms to a certain version of "reality".

I totally agree that people should listen to as much different HiFi equipment a possible to see what they like the sound of best. There is no substitute to actually listening with your own ears. This is just common sense.

 

But when it comes to things like replacement mains cables where in 99.9% of cases it will make no difference people should be aware of their own limitations when 'just listening'. In long term tests your auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play.

 

I agree that listening to equipment is of the upmost importance but with things like mains cables this is where methods such as instantaneous A/B switching come in useful during the selection process. When this is done the people who previously thought that they could hear subtle differences in long term sighted tests realise that when compared in instantaneous A/B switching tests there isn't really any difference in the sound between the mains cables. This has been proved many times and even you Cno admit to have never tried comparing cables in a blind test.

 

If anyone's interested why Cno was accused of scaremongering unnecessarily you can read it in the link below. Cno posted some incorrect information in post 15 on page 1 to which I replied.

http://www.whathifi.com/forum/hi-fi/just-put-in-some-mains-cable-upgrades

CnoEvil's picture
Offline
Last seen: 2 hours 42 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

steve_1979 wrote:

But when it comes to things like replacement mains cables where in 99.9% of cases it will make no difference people should be aware of their own limitations when 'just listening'. In long term tests your auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play.

I agree that listening to equipment is of the upmost importance but with things like mains cables this is where methods such as instantaneous A/B switching come in useful during the selection process. When this is done the people who previously thought that they could hear subtle differences in long term sighted tests realise that when compared in instantaneous A/B switching tests there isn't really any difference in the sound between the mains cables. This has been proved many times and even you Cno admit to have never tried comparing cables in a blind test.

If anyone's interested why Cno was accused of scaremongering unnecessarily you can read it in the link below. Cno posted some incorrect information in post 15 on page 1 to which I replied.

http://www.whathifi.com/forum/hi-fi/just-put-in-some-mains-cable-upgrades

Steve, I respect you enough not to fall out over this.

What I will say is, "Everyone" on here who has tried TQ Black, has heard a difference....that is so statistically significant, that it should not be dismissed so easily.

There are highly experienced members on here who are not gullable, and can hear the difference.....but they choose not to post, so as to avoid this exact situation. i would also suggest that the majority of people who give it a go, hear the difference........and this is despite the "nay brigade" telling them it's impossible.

My objection is not what you believe, or even that you tell people that it makes no difference...........but that you tell them not to try. I actually give people a little more credit than that. You are in effect telling them what is, and what is not worth listening to, which is tantamount to censorship....and all from a position where you have either tried very little, or not at all.

A cable dem by the likes of Chord or TCI might actually make you see things differently; but as Jack Nicholson said in a Few good men -  maybe "You can't handle the truth".  Wink

Now this is an ABX thread, so it's my last comment about cables.

 

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

chebby's picture
Offline
Last seen: 4 hours 28 min ago
Joined: 02/06/2008 - 09:40
Posts: 15876
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

...but as Jack Nicholson said in a Few good men -  maybe "You can't handle the truth".  Wink

And what happened to him?

"We are currently awaiting the loading of our complement of small lemon-soaked paper napkins for your comfort, refreshment and hygiene during the journey."

steve_1979's picture
Offline
Last seen: 1 week 4 days ago
Joined: 14/07/2010 - 21:04
Posts: 3791
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:
Steve, I respect you enough not to fall out over this.

That's something which I like and respect about you Cno. We're able to have a difference of opinion and discuss things sensibly without being rude or getting upset with each other and we're still able to be friends and have a laugh and joke afterwards.

:cheers:

 

 

CnoEvil wrote:
What I will say is, "Everyone" on here who has tried TQ Black, has heard a difference....that is so statistically significant, that it should not be dismissed so easily.

This is a perfect case example for this thread which is totally relevant here. I bet that 'everyone' who has tried TQ Black and heard a difference did so in a long term sighted comparison test. I'm not disregarding what these people say but under these comparison conditions your long term auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play too.

 

Now if they heard a difference in blind A/B comparisons or even passed an ABX test their claims would have far more credibility. We all know that long term sighted comparisons can be unreliable.

 

 

CnoEvil wrote:

My objection is not what you believe, or even that you tell people that it makes no difference...........but that you tell them not to try. I actually give people a little more credit than that. You are in effect telling them what is, and what is not worth listening to, which is tantamount to censorship....and all from a position where you have either tried very little, or none at all.

I have no objection with people listening to things for themselves. In fact quite the opposite I urge them to go and listen to as much hifi equipment as possible.

 

While it's best to do blind A/B comparisons wherever possible I also understand that in real life situations this is sometimes too inconvient to do. This is why people should to be aware of their own ears/brains/hearing/memory limitations when comparing products in long term sighted comparisons which are notoriously unreliable.

 

 

CnoEvil wrote:
but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth". Wink

Right back at yer Cno. Blum 3 Wink

TrevC's picture
Offline
Last seen: 8 hours 43 min ago
Joined: 12/06/2013 - 10:07
Posts: 788
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

What I find sad, is when all this obsession with testing stops people just going out and LISTENING. 

 

Not so. It tells you what you need to listen to. 

the record spot's picture
Offline
Last seen: 5 days 6 hours ago
Joined: 13/10/2007 - 14:36
Posts: 9252
RE: Subjective/objective testing /AB / AB/X, thoughts.

By somebody else's definition it would seem.

Craig M.'s picture
Offline
Last seen: 1 week 12 hours ago
Joined: 20/03/2008 - 20:10
Posts: 2871
RE: Subjective/objective testing /AB / AB/X, thoughts.

I'm probably an objectivist rather than a subjectivist when it comes to believing what is the best way to go about selecting hifi equipment, if the sound produced is the sole criteria.  I'm on the fence a little regarding AB/X though because I'm willing to accept that audio memory isn't fantastic, my own preferred choice would be just instantaneous switching between A and B without me knowing if B was just A being used again or if a switch had actually taken place, with the question just being can I hear any difference between A and B and if so then obviously personal preference enters the equation.  

 

While I'm happy to accept that audio memory can let a person down I do find it interesting that it doesn't seem so unreliable during sighted comparisons, when swapping between speaker cables (for instance) I've found it fairly easy to hear and remember differences and yet, using a rudimentary form of blind A/B switching, when I'm unaware of which cable is in use I've found it impossible to detect the same difference - or indeed any difference.  The other interesting finding was that after the blind comparison, returning to a sighted comparison the cables still sounded identical - the earlier sighted differences were no longer apparent.

 

I'm not sure the argument that 'I didn't expect to hear a difference but I did so it must be real' stands up, surely if a person didn't think there could be a difference they wouldn't have bothered making the comparison in the first place?  When I first got some interconnects to compare to the ones I already had I thought there probably wouldn't be a difference and felt a little silly asking the dealer if I could try some at home, and yet why was I going to the trouble of trying them?  Because a part of me obviously thought there might be something in it, even if it seemed silly.  I'm also not convinced that experience is any sort of guard against any sort of subconcious bias, there are a number of videos on youtube about something called the Mcgurk effect (were what you see directly influences what you hear), one of them by a chap who has been researching the Mcgurk effect for over 20 years - he is still fooled by it every time and there is nothing he can do to change that.

 

So that's my thought on the debate and my personal experience backs up the fact that, for me, my subjective opinion isn't especially reliable.  I can well understand why mags/websites/reviewers tend not to use level-matched blind testing though, 'it all sounds pretty much the same' isn't going to sell many mags is it? In fact, according to one very well regarded magazine editor, many audiophiles actively want there to be differences and the mag wouldn't sell if they didn't print them.

CnoEvil's picture
Offline
Last seen: 2 hours 42 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

Craig M. wrote:

I'm probably an objectivist rather than a subjectivist when it comes to believing what is the best way to go about selecting hifi equipment, if the sound produced is the sole criteria.  I'm on the fence a little regarding AB/X though because I'm willing to accept that audio memory isn't fantastic, my own preferred choice would be just instantaneous switching between A and B without me knowing if B was just A being used again or if a switch had actually taken place, with the question just being can I hear any difference between A and B and if so then obviously personal preference enters the equation.  

While I'm happy to accept that audio memory can let a person down I do find it interesting that it doesn't seem so unreliable during sighted comparisons, when swapping between speaker cables (for instance) I've found it fairly easy to hear and remember differences and yet, using a rudimentary form of blind A/B switching, when I'm unaware of which cable is in use I've found it impossible to detect the same difference - or indeed any difference.  The other interesting finding was that after the blind comparison, returning to a sighted comparison the cables still sounded identical - the earlier sighted differences were no longer apparent.

I'm not sure the argument that 'I didn't expect to hear a difference but I did so it must be real' stands up, surely if a person didn't think there could be a difference they wouldn't have bothered making the comparison in the first place?  When I first got some interconnects to compare to the ones I already had I thought there probably wouldn't be a difference and felt a little silly asking the dealer if I could try some at home, and yet why was I going to the trouble of trying them?  Because a part of me obviously thought there might be something in it, even if it seemed silly.  I'm also not convinced that experience is any sort of guard against any sort of subconcious bias, there are a number of videos on youtube about something called the Mcgurk effect (were what you see directly influences what you hear), one of them by a chap who has been researching the Mcgurk effect for over 20 years - he is still fooled by it every time and there is nothing he can do to change that.

So that's my thought on the debate and my personal experience backs up the fact that, for me, my subjective opinion isn't especially reliable.  I can well understand why mags/websites/reviewers tend not to use level-matched blind testing though, 'it all sounds pretty much the same' isn't going to sell many mags is it? In fact, according to one very well regarded magazine editor, many audiophiles actively want there to be differences and the mag wouldn't sell if they didn't print them.

I think that is a respectful, fair and reasonable approach....well most of it anyway.  Wink

We are all inclined to be drawn to, and influeneced by people who make sense of the world, in a way that ties in with how we naturally think. It is genuinly hard to remain open minded, as it is all too easy to reject anything that doesn't  fit conveniently with what we see as reality.......and I very much include myself in this.

As I age, I take a more tolerant, less "black and white" view of the world, and have found things are never quite as simple as I first thought.

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

CnoEvil's picture
Offline
Last seen: 2 hours 42 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

chebby wrote:

CnoEvil wrote:

...but as Jack Nicholson said in a Few good men -  maybe "You can't handle the truth".  Wink

And what happened to him?

Martyrdom isn't for everybody.  :shifty:

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

TrevC's picture
Offline
Last seen: 8 hours 43 min ago
Joined: 12/06/2013 - 10:07
Posts: 788
RE: Subjective/objective testing /AB / AB/X, thoughts.

busb wrote:

steve_1979 wrote:

DocG wrote:
OK, to put busb's remark somewhat sharper (may I, busb?): what if the participants all suffered neurosensory deafness ànd were really honest? Not a single positive result would be reported, so there is no difference, right? Utter :hs:, of course! I'm aware this is a daft example; it's just to make the point clear. Smile

This is a very good point that you and busb make. Smile ABX testing by definition has one very big limitation - it can only prove a positive result but it can't prove a negative one.

 

Basically if you really can hear a difference then you will get a positive result which categorically proves that you heard a difference. That's all fine and dandy and provides good quality useful data. However the problem with ABX testing is that you can't prove a negative result. If you can't hear any difference it doesn't prove that a difference can't be heard. It just shows that those people on that particular occasion didn't hear a difference. If they tried it again another time they might be able to hear a difference.

 

I fully agree with you that ABX testing isn't perfect. But it's still the most reliable comparison method available for checking whether or not you can hear any differences between two things. Don't forget that it's not just ABX which can't prove a negative result but all other listening comparison methods can't either.

 

I'd say ABX as a methodology is so flawed - it's almost pointless. 

 

So everything sounds the same? Really?

Pages

Log in or register to post comments