270 posts / 0 new
Last post
Macspur's picture
Offline
Last seen: 7 hours 31 min ago
Joined: 03/05/2010 - 06:56
Posts: 2089
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

busb wrote:

But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

I'm sitting here in a mixture of urine vapour, sweat and flies, while agitating my head lice.  :twisted:

Ah yuk! that's quite put me off my elevenses.

Smile

Electrocompaniet EMC1UP, Accuphase E350, Harbeth SHL5 Siltech 25th anniversary 33i XLR, Auditorium 23 SC, HiFi Racks Podium Reference rack.

busb's picture
Offline
Last seen: 2 weeks 2 days ago
Joined: 14/06/2011 - 20:08
Posts: 929
RE: Subjective/objective testing /AB / AB/X, thoughts.

fr0g wrote:

...and given that AB/X is mostly inpractical, yet AB/X tests are documented all over the internet, I wonder what makes many people simply dismiss the ones that have been done?

I am guessing that unless we personally take part, they don't really have much of a sway on us. Human pride? Is it that we need to "see for ourselves"?

 

An observation - personal experience is very powerful. Who can you trust if you cannot trust yourself? The knack is know when to trust oneself (through experience)!

The problem I have with DB ABX testing is the sneaking suspicion it doesn't work. Not only are they expensive to do properly, they are tedious to take part in (IMO). If they are not conducted with a great deal of care & preparation, many will not believe the results.

/as for ABX tests, I have 2 main concerns:

1.    How reliable is short-term memory, even with near instant switching? If it's not reliable, the testing is pointless, surely?

2.    The methodolgy is skewed. False positives are averaged out by statisical analysis - no problem there. People go very quiet when challenged over how false negatives should be handled.

An example. Some tests are carried out during a flu outburst. Many of the participants have severe colds that effect their hearing. Most cannot tell the difference between A & B. The conclusion: there ain't no diffeence between A & B! To me, this is not just bad science but very bad science! Until the whole method is shown to handle false responses equally, the resulting conclusions should be viewed with scepticism.

I'd love it if some method proved beyond doubt that all well-made cables sounded the same; that all amplifiers meeting a basic spec all sounded the same because I could then choose on the basis of looks, build quality, expected longevity of the manufacturer etc - all concrete, solid reasons.

Another point is how easily fooled our hearing is. Some people feel very insulted by the whole idea that they can be fooled. They shouldn't be! They just need to ackowledge the fact that their hearing can be fooled rather than is always being fooled. If our hearing was always being fooled, we would know if we were listening to music or a road drill!

Another phrase you will hear in these discussions is Expectation Bias. It does exist - even amongst music lovers! However, it's a term used indiscriminately & does not explain the situations where we expect to hear an "improvement" but don't. An example of expectation bias is when a group of listeners are asked to state their preference of something being tested which they duly do. They are then told that nothing was changed during the test. Many will feel slighted by such "dishonesty" - others learn from such an experience.

If we are willing to acknowledge our hearing is not so damn foolproof, we may just move the discussion on a bit without slashing each other as we all run around in the same circles.

MakkaPakka's picture
Offline
Last seen: 6 days 7 hours ago
Joined: 25/05/2013 - 16:44
Posts: 510
RE: Subjective/objective testing /AB / AB/X, thoughts.

busb wrote:

The problem I have with DB ABX testing is the sneaking suspicion it doesn't work. 

 

They seem to work just fine when it comes to speakers.

busb's picture
Offline
Last seen: 2 weeks 2 days ago
Joined: 14/06/2011 - 20:08
Posts: 929
RE: Subjective/objective testing ...

CnoEvil wrote:

busb wrote:

But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

I'm sitting here in a mixture of urine vapour, sweat and flies, while agitating my head lice.  :twisted:

Nit!  Wink

I'm using a sharp implement to remove the crust of grime from various parts of my anatomy. oh, my - must remove what feels like putty from my left armpit. But hey - what would you expect from someone who suffers from "Cognative Dissonance"!

:bounce:

CnoEvil's picture
Offline
Last seen: 7 hours 24 min ago
Joined: 21/08/2009 - 18:01
Posts: 12424
RE: Subjective/objective testing /AB / AB/X, thoughts.

Covenanter wrote:

CnoEvil wrote:

busb wrote:

But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

I'm sitting here in a mixture of urine vapour, sweat and flies, while agitating my head lice.  :twisted:

Does the "urine vapour" improve the soundstage though?

Chris

The soundstage is Pants.  :help:

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

CnoEvil's picture
Offline
Last seen: 7 hours 24 min ago
Joined: 21/08/2009 - 18:01
Posts: 12424
RE: Subjective/objective testing ...

busb wrote:

 - what would you expect from someone who suffers from "Cognative Dissonance"!

Hah.......I knew it!

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

steve_1979's picture
Offline
Last seen: 6 hours 48 min ago
Joined: 14/07/2010 - 21:04
Posts: 3885
RE: Subjective/objective testing /AB / AB/X, thoughts.

Macspur wrote:

CnoEvil wrote:

busb wrote:

But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

I'm sitting here in a mixture of urine vapour, sweat and flies, while agitating my head lice.  :twisted:

Ah yuk! that's quite put me off my elevenses.

Smile

:grin:

I've just read this while eating my breakfast (reheated chicken curry from the Chinese takeaway last night).

Hi-Fi - Teufel DecoderStation 5 > AVI DM5

Head-Fi - Epiphany Acoustics EHP-O2Di > Sennheisser HD700

Portable - Sony NWZ-A847 > Westone UM3x

busb's picture
Offline
Last seen: 2 weeks 2 days ago
Joined: 14/06/2011 - 20:08
Posts: 929
RE: Subjective/objective testing /AB / AB/X, thoughts.

MakkaPakka wrote:

busb wrote:

The problem I have with DB ABX testing is the sneaking suspicion it doesn't work. 

 

They seem to work just fine when it comes to speakers.

They also seem to work fine when asking people if they are listening to classical or pop music. My point being they only seem to work when the differences are extremely obvious. Many also argue that they don't work ever - even for speakers - stating that the quoted tests were not very scientific.

Feel free to address my point regarding false negatives though. I maybe entirely wrong & well-conducted DB ABX tests are indeed conclusive. Until someone does a little more than quote examples that validate their PoV without some scientific insight how the tests were conducted, I will remain sceptical.

CnoEvil's picture
Offline
Last seen: 7 hours 24 min ago
Joined: 21/08/2009 - 18:01
Posts: 12424
RE: Subjective/objective testing /AB / AB/X, thoughts.

steve_1979 wrote:

I've just read this while eating my breakfast (reheated chicken curry from the Chinese takeaway last night).

Well, you could be smelling of worse, before the day is out!    >)

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

MakkaPakka's picture
Offline
Last seen: 6 days 7 hours ago
Joined: 25/05/2013 - 16:44
Posts: 510
RE: Subjective/objective testing /AB / AB/X, thoughts.

busb wrote:

My point being they only seem to work when the differences are extremely obvious.

 

That's the whole point though - some people were claiming there are extremely obvious differences but another group of people disagreed and sought to challenge those claims.

People talk about night and day differences all the time which is at odds with all the test results. Actual test results are showing that any differences are very minor.

steve_1979's picture
Offline
Last seen: 6 hours 48 min ago
Joined: 14/07/2010 - 21:04
Posts: 3885
RE: Subjective/objective testing /AB / AB/X, thoughts.

busb wrote:

1. How reliable is short-term memory, even with near instant switching? If it's not reliable, the testing is pointless, surely?

While short term auditory memory is usually pretty good (especially with repetition) I have to also agree that it isn't perfect. However I do think that short term memory it's much better than long term memory. When it comes to makng A/B comparisons there are four options and all of them have pros and cons and non of them are perfect.

 

1. Making comparisons in any sighted tests. (IMO this can potentially be unreliable due to expectation bias)

 

2. Using your long term memory in blind comparisons and listening over long periods of time before swapping. (IMO this has okay reliabilty but will often lead to mistakes too)

 

3. Using your short term memory with an instantaneous A/B switching method. (IMO this has much better reliability but mistakes can still be made)

 

4. Using equipment to take scientific measurements. (IMO this is the best and most reliable method but only if the correct and relevant measurements are taken accurately and even then someone who is qualified is often required to correctly interprit the results without mistakes being made)

 

 

 

busb wrote:

2. The methodolgy is skewed. False positives are averaged out by statisical analysis - no problem there. People go very quiet when challenged over how false negatives should be handled.

In ABX tests if people just randomly think that they hear a difference when there isn't really one there then the results should come back almost exactly 50% correct and 50% incorrect. This is very straight forward IMO.

Hi-Fi - Teufel DecoderStation 5 > AVI DM5

Head-Fi - Epiphany Acoustics EHP-O2Di > Sennheisser HD700

Portable - Sony NWZ-A847 > Westone UM3x

DocG's picture
Offline
Last seen: 1 week 1 day ago
Joined: 01/05/2012 - 19:47
Posts: 1131
RE: Subjective/objective testing /AB / AB/X, thoughts.

steve_1979 wrote:

busb wrote:

2. The methodolgy is skewed. False positives are averaged out by statisical analysis - no problem there. People go very quiet when challenged over how false negatives should be handled.

In ABX tests if people just randomly think that they hear a difference when there isn't really one there then the results should come back almost exactly 50% correct and 50% incorrect. This is very straight forward IMO.

OK, to put busb's remark somewhat sharper (may I, busb?): what if the participants all suffered neurosensory deafness ànd were really honest? Not a single positive result would be reported, so there is no difference, right? Utter :hs:, of course! I'm aware this is a daft example; it's just to make the point clear. Smile

steve_1979's picture
Offline
Last seen: 6 hours 48 min ago
Joined: 14/07/2010 - 21:04
Posts: 3885
RE: Subjective/objective testing /AB / AB/X, thoughts.

DocG wrote:
OK, to put busb's remark somewhat sharper (may I, busb?): what if the participants all suffered neurosensory deafness ànd were really honest? Not a single positive result would be reported, so there is no difference, right? Utter :hs:, of course! I'm aware this is a daft example; it's just to make the point clear. Smile

This is a very good point that you and busb make. Smile ABX testing by definition has one very big limitation - it can only prove a positive result but it can't prove a negative one.

 

Basically if you really can hear a difference then you will get a positive result which categorically proves that you heard a difference. That's all fine and dandy and provides good quality useful data. However the problem with ABX testing is that you can't prove a negative result. If you can't hear any difference it doesn't prove that a difference can't be heard. It just shows that those people on that particular occasion didn't hear a difference. If they tried it again another time they might be able to hear a difference.

 

I fully agree with you that ABX testing isn't perfect. But it's still the most reliable comparison method available for checking whether or not you can hear any differences between two things. Don't forget that it's not just ABX which can't prove a negative result but all other listening comparison methods can't either.

Hi-Fi - Teufel DecoderStation 5 > AVI DM5

Head-Fi - Epiphany Acoustics EHP-O2Di > Sennheisser HD700

Portable - Sony NWZ-A847 > Westone UM3x

fr0g's picture
Offline
Last seen: 3 days 10 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2954
RE: Subjective/objective testing /AB / AB/X, thoughts.

I think one point is being missed here.

A single ABX test cn only prove a difference, not a lack of one. If we have a large enough pool of participants then it cn go some way to prove a lack of one.

And in either case, "prove" is not 100%, it's statistical probabilities to a smaller or larger extent.

TrevC's picture
Offline
Last seen: 1 week 6 days ago
Joined: 12/06/2013 - 10:07
Posts: 761
RE: Subjective/objective testing /AB / AB/X, thoughts.

Concentrate on speakers and possibly cartridges and the differences are enormous, so once you are using well made and appropriately gauged wire and interconnects and you have a powerful enough amp that is where you will get maximum value from any upgrade. ABX is something hifi magazines should use. It would get rid of the ridiculous mains lead "reviews" for starters, and I'll bet interconnects are not far behind.

Pages

Log in or register to post comments