270 posts / 0 new
Last post
fr0g's picture
Offline
Last seen: 2 days 2 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2954
Subjective/objective testing /AB / AB/X, thoughts.
English

I want some discussion on testing, whether it be purely subjective, purely objective or a mixture of the two.

 

I did try to have this discussion on another forum, but my posts were “moderated away” and the topic locked. Seemed there was to be no doubt at all cast on AB AB/X testing. Shame, it had stayed sensible and unheated, just some difference in opinion and misunderstanding of what was being said (IMO).

 

I'll start with where I stand. Many will have a clue already, but I fully approve of, and endorse the idea of blind testing, especially AB/X testing. AB/X is the only truly scientific method of comparing 2 similar sounding components (whether it be hardware, or software (in the case of codecs)).

 

My feeling is that if enough people do the same test, and can find no statistically meaningful difference between 2 components, then they are to all intents and purposes, the same (in terms of audibility).

 

With these tests I have proven (to myself) that it is worthless changing cables (of any description) beyond ensuring they are fit for the purpose...i.e. they aren't far too thin and high in resistance and they are well built and not prone to failure. I have also proven to myself that 256 Kbps and above, done properly in lossy MP3, AAC, OGG is more than enough to be transparent to the original and that anything over 16 /44.1 is pointless. Any changes in quality here are down to the original mastering.

 

So, while I think it is a small percentage of people, I believe AB/X testing results can be affected by the very fact that we “know” we are trying to test something, rather than simply sitting and enjoying the music.

 

However, I also think that single tests, or tests where there are not many participants can be flawed, flawed for the very same reasons as I think that non-blind AB testing is flawed, i.e. our brain plays tricks on us, and our ears hear things differently in different circumstances. Tests done with wine have shown that 2 identical wines, labelled differently, one cheap, one expensive, will receive different opinions by the same person. People will claim that the expensive one “tastes better”. But they are the same...However, the “expensive labelled version” does indeed taste better, as brain scans have shown more activity in the brain's pleasure centres whilst drinking it. So not only is the brain being fooled, it is reacting physically differently to 2 identical yet differently labelled bottles.

I think it's quite fair to extrapolate this to audio and say that 2 cables, one a super expensive one, one a cheap one can physically sound different to a person listening, even if they produce an identical result.

 

And example I gave (or tried to) was that my mother herself did not pass any exams at school, yet she was and is a clever person, however she was terrified of exams, and froze. Worry about "the future" I feel is no issue here as in those days, it wasn't so important for women to actually succeed in school either as most were expected to get married, have kids and stay at home.

Yes, this is an anecdotal case, and I have no idea if it is applicable, but it feels like it may be, at least to me.

 

So, on the one hand, I embrace blind testing, especially AB/X, but on the other say that they are rarely conclusive, and only begin to be with a large pool of participants.

 

Not only that, I can understand why some people don't bother. I reported a pass for 320 Kbps MP3, (which I think is remarkable myself), but the first response I got was “I would suspect the testing method” . I think it's responses such as that that make people think “why bother?”.

 

So, what do you people reckon?

Are they worth it to you?

Do you really trust your ears in a sighted test?

Do you think that the act of testing and knowing you are testing can alter the results?

Anything else?

 

 

 

 

 

Macspur's picture
Offline
Last seen: 13 hours 9 min ago
Joined: 03/05/2010 - 06:56
Posts: 2088
RE: Subjective/objective testing /AB / AB/X, thoughts.

Electrocompaniet EMC1UP, Accuphase E350, Harbeth SHL5 Siltech 25th anniversary 33i XLR, Auditorium 23 SC, HiFi Racks Podium Reference rack.

altruistic.lemon's picture
Offline
Last seen: 1 week 1 day ago
Joined: 25/07/2011 - 09:18
Posts: 1865
RE: Subjective/objective testing /AB / AB/X, thoughts.

There's long threads about this on a couple of other forums, many still going strong.

Do we need another of these stifling, and ultimately pointless in that consensus is never reached, threads?

the record spot's picture
Offline
Last seen: 2 months 5 days ago
Joined: 13/10/2007 - 14:36
Posts: 9252
RE: Subjective/objective testing /AB / AB/X, thoughts.

chebby's picture
Offline
Last seen: 47 min 7 sec ago
Joined: 02/06/2008 - 09:40
Posts: 16144
RE: Subjective/objective testing /AB / AB/X, thoughts.

What you are proposing is near impossible for most people without a stockroom full of gear, test equipment, volunteers and the time to do all the tests in a properly controlled fashion.

I think this should be a debate between you and professional testers / reviewers / manufacturers etc. and not the the average consumer or enthusiast.

I can't even stand auditioning gear in shops any more. (Headaches and 'ennui' set in rapidly so it's a hopeless task.)  The only way I can 'test' new equipment is to get an item home, decide if I like it and either buy it or return it.

Good luck if you do have the time, equipment, personnel, room and scientific rigour to do all this. I got weary just imagining it all when reading your post.

When / if I get to change my system next time around, I will ensure that the new system has absolutely everything I need in terms of functions and connectivity, make sure it is affordable, make sure it looks right and then try it at home. I will not be ABX'ing it against shelves full of alternatives (blind or otherwise) or involving listening panels of people who won't have to live with it like I do.

As I said before. This is a question for the industry and the reviewers.

"We are currently awaiting the loading of our complement of small lemon-soaked paper napkins for your comfort, refreshment and hygiene during the journey."

fr0g's picture
Offline
Last seen: 2 days 2 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2954
RE: Subjective/objective testing /AB / AB/X, thoughts.

Seems like people are not reading the whole post.

I am trying to say that while ABX is very useful, it is not 100% infallible. And I'd like to see opinions of where it is most useful, how it is most usefully implemented, etc.

 

No arguing or nastiness, simply discussion. Is it possible?

the record spot's picture
Offline
Last seen: 2 months 5 days ago
Joined: 13/10/2007 - 14:36
Posts: 9252
RE: Subjective/objective testing /AB / AB/X, thoughts.

Wasn't suggesting argument or nastiness fr0g, but it's been done to death.  Those that believe in it, have need of it or use it in their reviewing, buying or product development will use it, others trust their ears (see the Steve Hoffman forum for views on that for one example) and go down that route.

pauln's picture
Offline
Last seen: 6 hours 50 min ago
Joined: 26/02/2008 - 13:10
Posts: 700
RE: Subjective/objective testing /AB / AB/X, thoughts.

fr0g wrote:

No arguing or nastiness, simply discussion. Is it possible?

Probably not on this forum. Try somewhere more science based. This is predominantly a place for the 'audiophile' not the engineer. You won't be able to have a rational debate. There are some nice people posting on the headphone section though.

fr0g's picture
Offline
Last seen: 2 days 2 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2954
RE: Subjective/objective testing /AB / AB/X, thoughts.

the record spot wrote:

Wasn't suggesting argument or nastiness fr0g, but it's been done to death.  Those that believe in it, have need of it or use it in their reviewing, buying or product development will use it, others trust their ears (see the Steve Hoffman forum for views on that for one example) and go down that route.

 

The thing is, only the "My dad is bigger than yours" nastiness has been done to death. The "yes, no" tennis, the "you're dumb, no you are" nonsense.

There is no doubt that blind AB ABX can be useful, but I wanted a more philosophical (almost) take on it.

Asking too much ?

the record spot's picture
Offline
Last seen: 2 months 5 days ago
Joined: 13/10/2007 - 14:36
Posts: 9252
RE: Subjective/objective testing /AB / AB/X, thoughts.

pauln wrote:

fr0g wrote:

No arguing or nastiness, simply discussion. Is it possible?

Probably not on this forum. Try somewhere more science based. This is predominantly a place for the 'audiophile' not the engineer. You won't be able to have a rational debate. There are some nice people posting on the headphone section though.

 

The kind of dismissive comment about this place's membership that just serves to wind people up Paul.  You're welcome to your views, but there are several qualified people on here.  Some even tertiary qualified.  Others are regular guys who just want to chat about hifi or technology and others are new to the game.  Your problem is?  That we're not all scientists or bandy around the phrase "science" enough?  Such is life.

fr0g's picture
Offline
Last seen: 2 days 2 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2954
RE: Subjective/objective testing /AB / AB/X, thoughts.

pauln wrote:

fr0g wrote:

No arguing or nastiness, simply discussion. Is it possible?

Probably not on this forum. Try somewhere more science based. This is predominantly a place for the 'audiophile' not the engineer. You won't be able to have a rational debate. There are some nice people posting on the headphone section though.

 

I did first, posts were removed by an over-protective moderator with a rather fixed view on life.

the record spot's picture
Offline
Last seen: 2 months 5 days ago
Joined: 13/10/2007 - 14:36
Posts: 9252
RE: Subjective/objective testing /AB / AB/X, thoughts.

fr0g wrote:

the record spot wrote:

Wasn't suggesting argument or nastiness fr0g, but it's been done to death.  Those that believe in it, have need of it or use it in their reviewing, buying or product development will use it, others trust their ears (see the Steve Hoffman forum for views on that for one example) and go down that route.

 

The thing is, only the "My dad is bigger than yours" nastiness has been done to death. The "yes, no" tennis, the "you're dumb, no you are" nonsense.

There is no doubt that blind AB ABX can be useful, but I wanted a more philosophical (almost) take on it.

Asking too much ?

 

Probably not mate, but I'm with Chebby on this one.

fr0g's picture
Offline
Last seen: 2 days 2 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2954
RE: Subjective/objective testing /AB / AB/X, thoughts.

the record spot wrote:

pauln wrote:

fr0g wrote:

No arguing or nastiness, simply discussion. Is it possible?

Probably not on this forum. Try somewhere more science based. This is predominantly a place for the 'audiophile' not the engineer. You won't be able to have a rational debate. There are some nice people posting on the headphone section though.

 

The kind of dismissive comment about this place's membership that just serves to wind people up Paul.  You're welcome to your views, but there are several qualified people on here.  Some even tertiary qualified.  Others are regular guys who just want to chat about hifi or technology and others are new to the game.  Your problem is?  That we're not all scientists or bandy around the phrase "science" enough?  Such is life.

 

It is unnecessary, but then if we could get beyond that, I'd love to hear opinions on the subject in hand. Anyway, time to cook. Will my food be nicer than yesterday? Maybe I should start to save some and do a blind test each day Smile

altruistic.lemon's picture
Offline
Last seen: 1 week 1 day ago
Joined: 25/07/2011 - 09:18
Posts: 1865
RE: Subjective/objective testing /AB / AB/X, thoughts.

HDD audio?

chebby's picture
Offline
Last seen: 47 min 7 sec ago
Joined: 02/06/2008 - 09:40
Posts: 16144
RE: Subjective/objective testing /AB / AB/X, thoughts.

Reading through your post again, it is clear that you have proved to yourself (or had proved by others to your satisfaction) all the discussion points like differences (or lack of) between codecs, cables, levels of compression. wines, amplifiers etc.

Therefore any debate or disagreement (or differences in our experience) is going to go no-where because you've already been convinced. Your conviction in such matters are bound to lead to challenge and this will lead to arguments.

I don't drink wine and I don't believe in the efficacy of very expensive 'boutique' cables and my own (limited and entirely sighted) tests of compression rates have convinced me that 320K AAC VBR is the point where I struggle to tell the difference from Apple Lossless so that's what I use.

I can also tell the difference between 256K iTunes downloads and my own 256K AAC rips from CDs of the same tracks. (Rips are so obviously superior to my ears.)

I have no desire to prove or disprove this to others (or even discuss it). So if someone tells me I am wrong on all counts then good luck to you whoever you are. (£5 says it'll be Cnoevil anyway.)

 

"We are currently awaiting the loading of our complement of small lemon-soaked paper napkins for your comfort, refreshment and hygiene during the journey."

CnoEvil's picture
Online
Last seen: 5 hours 29 min ago
Joined: 21/08/2009 - 18:01
Posts: 12417
RE: Subjective/objective testing /AB / AB/X, thoughts.

pauln wrote:

There are some nice people posting on the headphone section though.

Well excuse us!

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

Pages

Log in or register to post comments