270 posts / 0 new
Last post
CnoEvil's picture
Offline
Last seen: 7 hours 22 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

fr0g wrote:

.I think many who do the "beating" (me occaisionally, when in the wrong mood) do it becuase they don't like to see people being misled or ripped off. I agree that sometimes the tone (actually most times) is all wrong, and more likely to cause someone to "dig in". Gentle persuasion imo is always best, and if someone "really" isn't interested then it's time to let go.

I would rather that people got both sides of the argument (preferably put respectfully) and were then left to make up their own mind, using whatever method they like.

Despite what many of the "Foo Fighters" would claim, there is (IMO) a genuine debate to be had. People like me, get slated for even suggesting that people try for themselves. I have done (and been to) extensive demos, where I heard clear differences, but since I can't prove it,  I never claim that this is anything other than my opinion.

I welcome the chalange of putiing my side of the debate......I'm confident enough of my position that I know there is a good chance that anyone prepared to try it, will hear the difference, if they choose the right product.

Almost every regular on here knows my position, so I won't bore you all with it any more (on this thread anyway).

FWIW. I usually have no problem with your forthright and mostly respectful approach.

 

 

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

matthewpiano's picture
Offline
Last seen: 6 hours 34 min ago
Joined: 23/11/2007 - 22:23
Posts: 8217
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

I suppose I at least owe you the courtesy of a relevant reply.  :shifty:

It is my opinion that if you have spent a lifetime (subjectively) listening to, and evaluating hifi, you have a reasonable likelihood of being more accurate identifying subtle changes, than someone who has walked into a hifi shop for the first time....though I would say that, wouldn't I.  :shifty:

I have managed to get through the rest of my life without ABX testing anything, and have come to trust my own judgement.....though I don't expect anyone else to, which is why I always recommend that people check for themselves.

FWIW. I have absolutely no problem with anyone using this test method if it makes them comfortable with their decision....what I'm less keen on is being constantly beaten over the head with it, by people who don't agree with me.  :wall:  :grin:

 

I agree with you entirely Cno.  I am perfectly happy to trust my own ears as they are all I have for enjoying music.  If I enjoy what I hear with my ears, then that is all I'm seeking.  I'm not terribly interested in the science of it, or even whether the sound is 100% accurate to the recording.  To explain the end of that statement - my opinion is that being 100% accurate to the recording isn't necessarily being 100% accurate to the music, and the thing that is most important to me, above all, is that the spirit, atmosphere and sound of the music is reproduced.

Like John, I'm not going to be so ridiculous as to suggest that my decision making is always 100% about sound either.  I believe other factors like how the product looks, how it feels in use, how it operates, what features it offers and sometimes even nostalgia all play a part in the total experience of owning and enjoying hi-fi equipment.  I suppose, if you want to make a more dispassionate choice which is based solely on the sound, ABX testing is probably the only way to do it but I would think this approach carries the risks of not fulfilling the other aspects of pride of ownership etc.

I couldn't agree more with your final sentence Cno and I feel exactly the same.

Rega RP3/Elys 2 - Graham Slee Gram Amp 1 - Rotel RCD965BX - Exposure 1010 amp - Dynaudio DM2/6 - Chord Co. Chameleon VEE3/Rumour 2 - Musical Fidelity V90-HPA - Sennheiser HD595

steve_1979's picture
Offline
Last seen: 1 week 4 days ago
Joined: 14/07/2010 - 21:04
Posts: 3791
RE: Subjective/objective testing /AB / AB/X, thoughts.

How many people here would like to have the option to use an instantaneous A/B switch box to help them make comparisons when demoing hifi equipment in shops?

matt49's picture
Offline
Last seen: 10 hours 27 min ago
Joined: 07/04/2013 - 12:07
Posts: 1952
RE: Subjective/objective testing /AB / AB/X, thoughts.

A couple of thoughts of a very abstract and inconclusive nature.

In other fields where blind testing is regularly used (e.g. pharmaceuticals, food and drink, cosmetics) with broadly similar aims (i.e. to establish which of a range of products is preferred or is more efficacious), the testers go to great lengths to establish a test environment  that's as neutral as possible. There are trained professionals who design and run such tests. It's become quite an industry, with its own standards and academic qualifications. Even with this intense academic focus on the neutrality of testing, and even with the huge amounts of money spent on it, the procedures and results of blind testing have been fiercely contested. (For an excellent and accessible introduction to the problems in testing psychopharmaceuticals: Irving Kirsch, The Emperor's New Drugs: Exploding the Antidepressant Myth, Basic Books, 2011.)

Whether hi-fi is interested in aspiring to these high standards or has the financial clout to do so, I very much doubt. Some big companies have done in-house blind testing in a relatively systematic way (e.g. Harman Kardon), and the developers of lossy codecs have used blind testing extensively. (As JD has already pointed out, switching between digital streams is relatively easy; switching between amps much less so.)

The screen door slams, Mary's dress waves ...

David@FrankHarvey's picture
Offline
Last seen: 6 hours 29 min ago
Joined: 27/06/2008 - 11:03
Posts: 11136
RE: Subjective/objective testing /AB / AB/X, thoughts.

altruistic.lemon wrote:
Tell me, David, what do you do for a living? Bit of a vested interest, really, wouldn't you agree?

As I'm not going to become a millionaire doing this, I'd hardly call it a 'vested interest'. I'd think differently if I was on £50k per year based on commission...

DavidF @FrankHarveyHiFi, Coventry.

"Long is the way, and hard, that out of hell leads up to light"

p_m_brown's picture
Offline
Last seen: 2 months 2 weeks ago
Joined: 01/02/2013 - 13:55
Posts: 301
RE: Subjective/objective testing /AB / AB/X, thoughts.

I must admit that I have disagreed with fr0g in a number of his posts as he has with mine.

But, I do think that a/b testing is important when buying new equipment. I have done it with almost all of the components and cables that I have bought over the years. However, blind testing is IMHO, not practical or really possible in the domestic environment or at least in MY domestic environment. I have mulled over the differences between amps and dacs by switching components and trusting my ears to make an informed decision. I do not find myself swayed by hype, or taking other people's opinions as gospel, but if I demo an item that makes a noticeable improvement to my system then I trust my ears and spend my money. 

In summary, I suppose I try to select subjectively, as accurately as possible! That goes for hifi and wine  :cheers:

fr0g's picture
Offline
Last seen: 20 hours 26 min ago
Joined: 07/01/2008 - 18:38
Posts: 2936
RE: Subjective/objective testing /AB / AB/X, thoughts.

I think one thing that has been shown to me here, is that if a discussion point that often causes conflict, is taken out and discussed on its own merits is less likely to lead to the squabbling we often see. We all have opinions, and nobody is "right".

I do think in some circumstances, blind testing, esp AB/X, is incredibly useful, if only to prove something to yourself, for or against, but in the end I also agree that ultimately, all we can do is sit down and listen, and ask ourselves "Is this nice or what??"

 

Cheers folks.

davedotco's picture
Offline
Last seen: 1 day 8 hours ago
Joined: 24/04/2013 - 10:02
Posts: 3073
RE: Subjective/objective testing /AB / AB/X, thoughts.

Busy day at the cricket, so have only just got in and read the thread. I have a number of thoughts on the subject, not entirely consistent but hear goes.

Taking part in almost any reasonably well conducted blind test is extremely instructive for any enthusiast, I am not necessarily suggesting you buy your equipment on the outcome of such tests, just that they can give you a very swift grounding of just how small differences can be.

I feel that an experienced listener can, over a period, make judgements on equipment that goes beyond what can be learned from blind testing, that said I think you need to have a lot of experience to make those calls.

One of the few people who have this experience is the dealer, someone good can save you a lot of time and hassle, probably money too. Sadly there are few such that opperate at the budget end of the price spectrum.

Don't try to hard to discount or try to overcome bias, sometimes you can use it to your advantage, buying a better component because you prefer it for some reason other than pure sound quality, can increase your satisfaction by making you more 'comfortable' with it.

For me it really is not an either/or situation, simply being aware of some of the realities of the way hi-fi equipment works and is demonstrated can help you make better informed decisions, that for me is the biggest advantage that objective knowledge can bring to subjective listening.

Anonymous
Anonymous's picture
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

pauln wrote:

There are some nice people posting on the headphone section though.

Well excuse us!

They are nice over their Cno 

fr0g's picture
Offline
Last seen: 20 hours 26 min ago
Joined: 07/01/2008 - 18:38
Posts: 2936
RE: Subjective/objective testing /AB / AB/X, thoughts.

John Duncan wrote:

 

I think this is an interesting question, which deserves some discussion.  I don't believe fr0g is trying to start a fight (though no doubt one will start anyway). It is my opinion that this is a subjectivist forum.  In other words, I think people who come here and stay here take the view that the best way of judging a piece of audio equipment is by listening to it.  I share that view, but am not immune to the idea of more scientific testing (of which listening is, I believe, not an example, bit more of that later). I think 'formal testing' by means of, for example, ABX is about the best way of deciding whether one thing is different from another (notice that I do not use the word 'better').  However, the ease by which one can achieve this can vary enormously, depending on what it is that one is trying to ABX.  For example, I think the tools exist whereby one can satisfy oneself quite easily of the superiority or otherwise of, say higher resolution audio over standard CD or 320k or lower resolution compressed files.  I have never tried the Foobar ABX plugin, primarily because I've managed to satisfy myself by more prosaic means (listening) that they are pretty much functionally equivalent (for my purposes).  I'd urge people to try it, simply because it's an easy way of taking a position on how one feels about what is 'good enough' from a source point of view. However, I think when it comes to individual pieces of audio equipment, it's much harder to do.  All you need to do is look at Alan Shaw's Harbeth Challenge to see the angst that it has engendered in so many quarters.  Personally, I think that challenge was never going to be answered (let alone met or beaten) because it was not about hifi equipment at all really, but any respondent's ability to come up with an instantaneous switching mechanism so that an ABX test could be carried out against equipment that doesn't really lend itself to the practice.  That, combined with the fact that several 'caveats' about equipment's behaviour were put in place meant that it veered too much towards the old 'all amps sound the same provided you stick a graphic EQ in line with them' question. And there is the nub of the problem - I think it's actually difficult, if not impossible, to ABX hifi equipment (without starting to add other things into the mix like comparators and suchlike, which immediately bring claims of 'signal degradation' which invalidate the tests).  It takes too long to swap out pieces of equipment to get round the 'aural memory' issue which pure objectivists claim is the reason behind all sorts of reported phenomena, such as burn-in.  And TBH, I'd argue that sticking a sheet over a speaker so you don't know what it is probably counterproductive. So in absence of a practical, repeatable test method, I'm inclined to go with what's left, which is listening.  Obviously we can try to remove variables which can affect the result, such as levels, but other than that when I'm looking for a new piece of kit I'm inclined to consider very few questions: - Does it sound better TO ME (since nobody else is buying it for me)? - Is it more functional? - Does it look better? In other words, if I want to spend my money, then I shall, but I'm not so stupid as to declare that the only thing which influences me in that decision is sound quality (since for the last 20 years I'd argue that improvements have been marginal at best, though the cost to implement has decreased fairly inversely to Moore's Law).  I will also argue that until somebody comes up with a foolproof, easy method of proving to me (in a nice environment with good coffee, as opposed to a lab or a shed) that X is 1,000 times better than Y I will, as they are forced to do (and are - somewhat curiously - happy to do), have to make do with listening and my own, foolish, subjective opinion.

 

I think this is an interesting question, which deserves some discussion.  I don't believe fr0g is trying to start a fight (though no doubt one will start anyway).

It is my opinion that this is a subjectivist forum.  In other words, I think people who come here and stay here take the view that the best way of judging a piece of audio equipment is by listening to it.  I share that view, but am not immune to the idea of more scientific testing (of which listening is, I believe, not an example, bit more of that later).

I think 'formal testing' by means of, for example, ABX is about the best way of deciding whether one thing is different from another (notice that I do not use the word 'better').  However, the ease by which one can achieve this can vary enormously, depending on what it is that one is trying to ABX.  For example, I think the tools exist whereby one can satisfy oneself quite easily of the superiority or otherwise of, say higher resolution audio over standard CD or 320k or lower resolution compressed files.  I have never tried the Foobar ABX plugin, primarily because I've managed to satisfy myself by more prosaic means (listening) that they are pretty much functionally equivalent (for my purposes).  I'd urge people to try it, simply because it's an easy way of taking a position on how one feels about what is 'good enough' from a source point of view.

However, I think when it comes to individual pieces of audio equipment, it's much harder to do.  All you need to do is look at Alan Shaw's Harbeth Challenge to see the angst that it has engendered in so many quarters.  Personally, I think that challenge was never going to be answered (let alone met or beaten) because it was not about hifi equipment at all really, but any respondent's ability to come up with an instantaneous switching mechanism so that an ABX test could be carried out against equipment that doesn't really lend itself to the practice.  That, combined with the fact that several 'caveats' about equipment's behaviour were put in place meant that it veered too much towards the old 'all amps sound the same provided you stick a graphic EQ in line with them' question.

And there is the nub of the problem - I think it's actually difficult, if not impossible, to ABX hifi equipment (without starting to add other things into the mix like comparators and suchlike, which immediately bring claims of 'signal degradation' which invalidate the tests).  It takes too long to swap out pieces of equipment to get round the 'aural memory' issue which pure objectivists claim is the reason behind all sorts of reported phenomena, such as burn-in.  And TBH, I'd argue that sticking a sheet over a speaker so you don't know what it is probably counterproductive.

So in absence of a practical, repeatable test method, I'm inclined to go with what's left, which is listening.  Obviously we can try to remove variables which can affect the result, such as levels, but other than that when I'm looking for a new piece of kit I'm inclined to consider very few questions:

- Does it sound better TO ME (since nobody else is buying it for me)?

- Is it more functional?

- Does it look better?

In other words, if I want to spend my money, then I shall, but I'm not so stupid as to declare that the only thing which influences me in that decision is sound quality (since for the last 20 years I'd argue that improvements have been marginal at best, though the cost to implement has decreased fairly inversely to Moore's Law).  I will also argue that until somebody comes up with a foolproof, easy method of proving to me (in a nice environment with good coffee, as opposed to a lab or a shed) that X is 1,000 times better than Y, I will, as they are forced to do (and seem - somewhat curiously - happy to do), have to make do with listening and my own, foolish, subjective opinion.

[written in one go with no proof reading.  I may change my opinions at a later date]

 

Good post, and I tend to agree for the most part.

AB/X is desirable, but inpractical in most situations, other than those that only involve software.

I feel Steve's reply is important too ( mentioning a switchbox).

While AB/X is next to impossible to practically do in a demo, at least being given the chance to blind A/B would be very useful. I wonder how many dealers offer this?

 

And when it comes down to traditional, passive speaker based systems, I personally feel that most equipment (other than the speakers) are at such a level of competency as to bring the buying decision more down to looks, brand loyalty or attraction, than anything else (I am not saying that there are no differences btw, just that they are smaller these days than at any time in history)

 

fr0g's picture
Offline
Last seen: 20 hours 26 min ago
Joined: 07/01/2008 - 18:38
Posts: 2936
RE: Subjective/objective testing /AB / AB/X, thoughts.

...and given that AB/X is mostly inpractical, yet AB/X tests are documented all over the internet, I wonder what makes many people simply dismiss the ones that have been done?

I am guessing that unless we personally take part, they don't really have much of a sway on us. Human pride? Is it that we need to "see for ourselves"?

 

 

 

CnoEvil's picture
Offline
Last seen: 7 hours 22 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

andybeau wrote:

CnoEvil wrote:

pauln wrote:

There are some nice people posting on the headphone section though.

Well excuse us!

They are nice over their Cno 

:grin:

I must get me some headphones then.

My last ones were large, black, Pioneer over-ear jobbies, bought in 1977....and now lie unloved (and in bits), in a drawer somewhere.

IIRC they were these ones: http://www.ebay.com/itm/Vintage-Pioneer-SE-205-Stereo-Headphones-Great-Sound-/261212276885?pt=Vintage_Electronics_R2&hash=item3cd1770895

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

busb's picture
Offline
Last seen: 1 week 5 days ago
Joined: 14/06/2011 - 20:08
Posts: 928
RE: Subjective/objective testing /AB / AB/X, thoughts.

the record spot wrote:

pauln wrote:

fr0g wrote:

No arguing or nastiness, simply discussion. Is it possible?

Probably not on this forum. Try somewhere more science based. This is predominantly a place for the 'audiophile' not the engineer. You won't be able to have a rational debate. There are some nice people posting on the headphone section though.

 

The kind of dismissive comment about this place's membership that just serves to wind people up Paul.  You're welcome to your views, but there are several qualified people on here.  Some even tertiary qualified.  Others are regular guys who just want to chat about hifi or technology and others are new to the game.  Your problem is?  That we're not all scientists or bandy around the phrase "science" enough?  Such is life.

 

I have to agree. Some people use so-called science to bash other people with. The result? They alienate & further polarise many away from rational debate due to arrogance. No one is forced to post here. Don't like it? Vote with your feet. But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

CnoEvil's picture
Offline
Last seen: 7 hours 22 min ago
Joined: 21/08/2009 - 18:01
Posts: 12074
RE: Subjective/objective testing /AB / AB/X, thoughts.

busb wrote:

But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

I'm sitting here in a mixture of urine vapour, sweat and flies, while agitating my head lice.  :twisted:

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

Covenanter's picture
Offline
Last seen: 20 hours 19 min ago
Joined: 20/07/2012 - 10:16
Posts: 1225
RE: Subjective/objective testing /AB / AB/X, thoughts.

CnoEvil wrote:

busb wrote:

But wait - some have a duty to "truth" & selflessly have to educate the unwashed.

I'm sitting here in a mixture of urine vapour, sweat and flies, while agitating my head lice.  :twisted:

Does the "urine vapour" improve the soundstage though?

Chris

Marantz PM8005 / SA8005 / KEF R700s / AKG K702

Pages

Log in or register to post comments