270 posts / 0 new
Last post
MakkaPakka's picture
Offline
Last seen: 2 months 3 weeks ago
Joined: 25/05/2013 - 16:44
Posts: 508
RE: Subjective/objective testing /AB / AB/X, thoughts.

My point was really that people are sweating the small details when they haven't even thought about the big details.

I generally agree with you on ABX testing - if people are claiming night and day differences between things then the tests would all be passed easily just as they are with speakers. If the tests are not being passed then the differences are not likely to be worth spending a lot of cash one.

I recommend people try www.mp3ornot.com just to get an idea of how difficult blind testing is. You go into it assuming it will be a breeze but it isn't.

 

altruistic.lemon's picture
Offline
Last seen: 2 weeks 4 days ago
Joined: 25/07/2011 - 09:18
Posts: 1865
RE: Subjective/objective testing /AB / AB/X, thoughts.

David@FrankHarvey wrote:

Like I said, some will understand what I said, some won't Smile

(and I DID say that I fully understand the need for volume matching!)

Oh, I understand alright, don't you worry about that!
Supreme's picture
Offline
Last seen: 2 months 3 weeks ago
Joined: 25/06/2013 - 15:44
Posts: 157
RE: Subjective/objective testing /AB / AB/X, thoughts.

A while back me and the family went 10 pin bowling in Torquay. Outside the bowling alley I could hear a nasty high pitch noise and mentioned this to the guy at the desk. He told me it was to keep kids/teens from hanging around outside, the high pitch noise, very high yet faint was designed so that only the young could hear it, annoy them, and they would hopefully move on. Well, I'm 45 and I can hear it clearly but the wife didn't.

My point is that some people can hear pitches, tones etc that others can't. I'm sure I don't need to spell the rest out.

John Duncan's picture
Offline
Last seen: 50 min 20 sec ago
Joined: 08/01/2008 - 17:25
Posts: 22993
RE: Subjective/objective testing /AB / AB/X, thoughts.

fr0g wrote:

altruistic.lemon wrote:

HDD audio?

 

I have no intention of naming the forum. At last count I was a member on at least 6, only 2 as fr0g. There are overzealous admins on most.

Whereas here, in 'my little fiefdom', I predominantly can't be EDITED.

CnoEvil's picture
Offline
Last seen: 29 min 23 sec ago
Joined: 21/08/2009 - 18:01
Posts: 12105
RE: Subjective/objective testing /AB / AB/X, thoughts.

I suppose I at least owe you the courtesy of a relevant reply.  :shifty:

It is my opinion that if you have spent a lifetime (subjectively) listening to, and evaluating hifi, you have a reasonable likelihood of being more accurate identifying subtle changes, than someone who has walked into a hifi shop for the first time....though I would say that, wouldn't I.  :shifty:

I have managed to get through the rest of my life without ABX testing anything, and have come to trust my own judgement.....though I don't expect anyone else to, which is why I always recommend that people check for themselves.

FWIW. I have absolutely no problem with anyone using this test method if it makes them comfortable with their decision....what I'm less keen on is being constantly beaten over the head with it, by people who don't agree with me.  :wall:  :grin:

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

fr0g's picture
Offline
Last seen: 1 day 12 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2940
RE: Subjective/objective testing /AB / AB/X, thoughts.

David@FrankHarvey wrote:

Quote:
However, I also think that single tests, or tests where there are not many participants can be flawed, flawed for the very same reasons as I think that non-blind AB testing is flawed, i.e. our brain plays tricks on us, and our ears hear things differently in different circumstances. Tests done with wine have shown that 2 identical wines, labelled differently, one cheap, one expensive, will receive different opinions by the same person. People will claim that the expensive one “tastes better”. But they are the same...However, the “expensive labelled version” does indeed taste better, as brain scans have shown more activity in the brain's pleasure centres whilst drinking it. So not only is the brain being fooled, it is reacting physically differently to 2 identical yet differently labelled bottles.
But is this by 'ordinary' newbies to wine tasting, or by 'professionals' who are up their own backside and like the sound of their own voice?

Quote:
Are they worth it to you?
Only if they are properly conducted, but there's so many aspects that can be 'fudged' that you don't know if you're trying everything fairly.

Quote:
Do you really trust your ears in a sighted test?
I think some people can. Yes, I agree that there is a lot of bias expectation, but I do believe that people can rise above it and 'think for themselves'. In an ABX test, it doesn't really matter as it is generally a bit of fun, but I think it is different when people are actually spending their own money - if someone is ABX testing two interconnects for £30 and £3000, those not buying may well say the more expensive one "is better", but those doing it with a view to possibly spending £3000 are going to treat things a little more seriously. You could say that those who can afford a £3k interconnect may well choose the better one because they can, but on the flip side, the majority of people who are into this hobby generally can't afford £3k cables (no disrespect intended to anyone - I include myself in that generalisation!) are going to be a little more careful with their extremely hard earned cash.

Quote:
Do you think that the act of testing and knowing you are testing can alter the results?
Yes.

 

Something you said on 'that other forum':

Quote:
I'm not claiming long-term testing is better. And while I stand by what I said (above), I think it's more than "stress", rather different conditions.You listen at home in comfort, relaxing, with your mind on nothing but the music. You then start playing (30?) second snaps of the music back and forth. It isn't the same. You "will" hear it differently.

Most of my impressions of equipment nowadays tend to be based on long term listening. As long as you know your favourite albums well, I don't see any problems. Sometimes you hear something in a track like the bassline or the drums that just sound amazing on a particular system, but less impressive on others. My preference of Chord's Epic over Odyssey with the KEF Blades and an accompanying Cyrus system was based on long term listening. With the Epic, the system always sounded great. We changed to four core Odyssey, and over the following weeks, the system just seemed to lose something - it just didn't sound as good. Putting the Epic back on, and the following few weeks the system seemed to sound better again.

 

There's also the aspect of ABX testing that many tend to push - volume matching. While I fully understand and appreciate the need for this (as well as the effects of not level matching), I'm not so sure that it plays such a part as people think. What I mean is, we all have a favourite listening level. It doesn't matter what album you put on, or how different the level is between them, most people will adjust the volume to that level. I'd guess that most of us would probably get two different systems playing the same thing to within 1dB when setting our own listening levels. As an example, even when my neighbour goes out, I don't play my system any louder - I still listen at the same volume I normally do, because I find it comfortable for me and the room. I'm not expecting this last point to be popular, maybe not even understood, but I've said anyway. So there Smile

 

 

Thank you David, I don't necessarily agree with all you have said, but that was the best and most reasonable reply to this issue yet. Congrats. Smile

David@FrankHarvey's picture
Offline
Last seen: 11 hours 10 min ago
Joined: 27/06/2008 - 11:03
Posts: 11158
RE: Subjective/objective testing /AB / AB/X, thoughts.

Thanks fr0g - your reply is enough to justify me spending the time it took to write it.

I don't really care if no one agrees with me - I won't be the one getting my knickers in a twist about it   Smile

DavidF @FrankHarveyHiFi, Coventry.

"Long is the way, and hard, that out of hell leads up to light"

fr0g's picture
Offline
Last seen: 1 day 12 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2940
RE: Subjective/objective testing /AB / AB/X, thoughts.

 

CnoEvil wrote:

I suppose I at least owe you the courtesy of a relevant reply.  :shifty:

It is my opinion that if you have spent a lifetime (subjectively) listening to, and evaluating hifi, you have a reasonable likelihood of being more accurate identifying subtle changes, than someone who has walked into a hifi shop for the first time....though I would say that, wouldn't I.  :shifty:

 

I think this could be correct, but I do wonder if one could ever be truly unbiased with the knowledge of what they are listening to .

 

CnoEvil wrote:

I have managed to get through the rest of my life without ABX testing anything, and have come to trust my own judgement.....though I don't expect anyone else to, which is why I always recommend that people check for themselves.

FWIW. I have absolutely no problem with anyone using this test method if it makes them comfortable with their decision....what I'm less keen on is being constantly beaten over the head with it, by people who don't agree with me.  :wall:  :grin:

 

I think many who do the "beating" (me occaisionally, when in the wrong mood) do it becuase they don't like to see people being misled or ripped off. I agree that sometimes the tone (actually most times) is all wrong, and more likely to cause someone to "dig in". Gentle persuasion imo is always best, and if someone "really" isn't interested then it's time to let go.

CnoEvil's picture
Offline
Last seen: 29 min 23 sec ago
Joined: 21/08/2009 - 18:01
Posts: 12105
RE: Subjective/objective testing /AB / AB/X, thoughts.

David@FrankHarvey wrote:

Like I said, some will understand what I said, some won't Smile

I understand.......but I need help!  :twisted:

"We should no more let numbers define audio quality than we should let chemical analysis be the arbiter of fine wines."  Nelson Pass

fr0g's picture
Offline
Last seen: 1 day 12 hours ago
Joined: 07/01/2008 - 18:38
Posts: 2940
RE: Subjective/objective testing /AB / AB/X, thoughts.

David@FrankHarvey wrote:

I don't really care if no one agrees with me - I won't be the one getting my knickers in a twist about it   Smile

 

I think everyone should heed exactly this "advice".

altruistic.lemon's picture
Offline
Last seen: 2 weeks 4 days ago
Joined: 25/07/2011 - 09:18
Posts: 1865
RE: Subjective/objective testing /AB / AB/X, thoughts.

Tell me, David, what do you do for a living? Bit of a vested interest, really, wouldn't you agree?

steve_1979's picture
Offline
Last seen: 2 weeks 2 days ago
Joined: 14/07/2010 - 21:04
Posts: 3791
RE: Subjective/objective testing /AB / AB/X, thoughts.

Duplicate post.

steve_1979's picture
Offline
Last seen: 2 weeks 2 days ago
Joined: 14/07/2010 - 21:04
Posts: 3791
RE: Subjective/objective testing /AB / AB/X, thoughts.

fr0g wrote:
...I can understand why some people don't bother. I reported a pass for 320 Kbps MP3, (which I think is remarkable myself), but the first response I got was “I would suspect the testing method” . I think it's responses such as that that make people think “why bother?”.

Regarding your claims that some people can successfully pass a 320kbps MP3 vs lossless ABX test I said that if the tests were done properly and in a non-biased way then that would be fair enough. I'm not saying they're wrong I'm just saying is that I'd question the details if they managed to pass this particular ABX test.

 

While I'm not saying that these people are definitely wrong I do think that all things considered it seems more likely to be that either the methodology of their testing is somehow flawed or they're just blatantly lying to win a forum argument. Would you agree or disagree with this fr0g?

 

(All said with a big friendly smile on my face Biggrin )

John Duncan's picture
Offline
Last seen: 50 min 20 sec ago
Joined: 08/01/2008 - 17:25
Posts: 22993
RE: Subjective/objective testing /AB / AB/X, thoughts.

 

I think this is an interesting question, which deserves some discussion.  I don't believe fr0g is trying to start a fight (though no doubt one will start anyway). It is my opinion that this is a subjectivist forum.  In other words, I think people who come here and stay here take the view that the best way of judging a piece of audio equipment is by listening to it.  I share that view, but am not immune to the idea of more scientific testing (of which listening is, I believe, not an example, bit more of that later). I think 'formal testing' by means of, for example, ABX is about the best way of deciding whether one thing is different from another (notice that I do not use the word 'better').  However, the ease by which one can achieve this can vary enormously, depending on what it is that one is trying to ABX.  For example, I think the tools exist whereby one can satisfy oneself quite easily of the superiority or otherwise of, say higher resolution audio over standard CD or 320k or lower resolution compressed files.  I have never tried the Foobar ABX plugin, primarily because I've managed to satisfy myself by more prosaic means (listening) that they are pretty much functionally equivalent (for my purposes).  I'd urge people to try it, simply because it's an easy way of taking a position on how one feels about what is 'good enough' from a source point of view. However, I think when it comes to individual pieces of audio equipment, it's much harder to do.  All you need to do is look at Alan Shaw's Harbeth Challenge to see the angst that it has engendered in so many quarters.  Personally, I think that challenge was never going to be answered (let alone met or beaten) because it was not about hifi equipment at all really, but any respondent's ability to come up with an instantaneous switching mechanism so that an ABX test could be carried out against equipment that doesn't really lend itself to the practice.  That, combined with the fact that several 'caveats' about equipment's behaviour were put in place meant that it veered too much towards the old 'all amps sound the same provided you stick a graphic EQ in line with them' question. And there is the nub of the problem - I think it's actually difficult, if not impossible, to ABX hifi equipment (without starting to add other things into the mix like comparators and suchlike, which immediately bring claims of 'signal degradation' which invalidate the tests).  It takes too long to swap out pieces of equipment to get round the 'aural memory' issue which pure objectivists claim is the reason behind all sorts of reported phenomena, such as burn-in.  And TBH, I'd argue that sticking a sheet over a speaker so you don't know what it is probably counterproductive. So in absence of a practical, repeatable test method, I'm inclined to go with what's left, which is listening.  Obviously we can try to remove variables which can affect the result, such as levels, but other than that when I'm looking for a new piece of kit I'm inclined to consider very few questions: - Does it sound better TO ME (since nobody else is buying it for me)? - Is it more functional? - Does it look better? In other words, if I want to spend my money, then I shall, but I'm not so stupid as to declare that the only thing which influences me in that decision is sound quality (since for the last 20 years I'd argue that improvements have been marginal at best, though the cost to implement has decreased fairly inversely to Moore's Law).  I will also argue that until somebody comes up with a foolproof, easy method of proving to me (in a nice environment with good coffee, as opposed to a lab or a shed) that X is 1,000 times better than Y I will, as they are forced to do (and are - somewhat curiously - happy to do), have to make do with listening and my own, foolish, subjective opinion.

 

I think this is an interesting question, which deserves some discussion.  I don't believe fr0g is trying to start a fight (though no doubt one will start anyway).

It is my opinion that this is a subjectivist forum.  In other words, I think people who come here and stay here take the view that the best way of judging a piece of audio equipment is by listening to it.  I share that view, but am not immune to the idea of more scientific testing (of which listening is, I believe, not an example, bit more of that later).

I think 'formal testing' by means of, for example, ABX is about the best way of deciding whether one thing is different from another (notice that I do not use the word 'better').  However, the ease by which one can achieve this can vary enormously, depending on what it is that one is trying to ABX.  For example, I think the tools exist whereby one can satisfy oneself quite easily of the superiority or otherwise of, say higher resolution audio over standard CD or 320k or lower resolution compressed files.  I have never tried the Foobar ABX plugin, primarily because I've managed to satisfy myself by more prosaic means (listening) that they are pretty much functionally equivalent (for my purposes).  I'd urge people to try it, simply because it's an easy way of taking a position on how one feels about what is 'good enough' from a source point of view.

However, I think when it comes to individual pieces of audio equipment, it's much harder to do.  All you need to do is look at Alan Shaw's Harbeth Challenge to see the angst that it has engendered in so many quarters.  Personally, I think that challenge was never going to be answered (let alone met or beaten) because it was not about hifi equipment at all really, but any respondent's ability to come up with an instantaneous switching mechanism so that an ABX test could be carried out against equipment that doesn't really lend itself to the practice.  That, combined with the fact that several 'caveats' about equipment's behaviour were put in place meant that it veered too much towards the old 'all amps sound the same provided you stick a graphic EQ in line with them' question.

And there is the nub of the problem - I think it's actually difficult, if not impossible, to ABX hifi equipment (without starting to add other things into the mix like comparators and suchlike, which immediately bring claims of 'signal degradation' which invalidate the tests).  It takes too long to swap out pieces of equipment to get round the 'aural memory' issue which pure objectivists claim is the reason behind all sorts of reported phenomena, such as burn-in.  And TBH, I'd argue that sticking a sheet over a speaker so you don't know what it is probably counterproductive.

So in absence of a practical, repeatable test method, I'm inclined to go with what's left, which is listening.  Obviously we can try to remove variables which can affect the result, such as levels, but other than that when I'm looking for a new piece of kit I'm inclined to consider very few questions:

- Does it sound better TO ME (since nobody else is buying it for me)?

- Is it more functional?

- Does it look better?

In other words, if I want to spend my money, then I shall, but I'm not so stupid as to declare that the only thing which influences me in that decision is sound quality (since for the last 20 years I'd argue that improvements have been marginal at best, though the cost to implement has decreased fairly inversely to Moore's Law).  I will also argue that until somebody comes up with a foolproof, easy method of proving to me (in a nice environment with good coffee, as opposed to a lab or a shed) that X is 1,000 times better than Y, I will, as they are forced to do (and seem - somewhat curiously - happy to do), have to make do with listening and my own, foolish, subjective opinion.

[written in one go with no proof reading.  I may change my opinions at a later date]

steve_1979's picture
Offline
Last seen: 2 weeks 2 days ago
Joined: 14/07/2010 - 21:04
Posts: 3791
RE: Subjective/objective testing /AB / AB/X, thoughts.

Good post JD. Smile

 

That is a good point. It's easy to ABX music files but it is a more fiddly to do with HiFi equipment.

 

I have done simple A/B comparisons in pro-audio shops where it's straight forward and easy to switch between two sets of active speakers or two sets of DACs using one of their mixers or similar equipment to do the switching.

But I've never been to a HiFi shop where they allow you to use a simple switch box to allow you to do quick comparisons between various pieces of HiFi equipment though. A switch box is fairly cheap and easy to setup, it's a shame that more HiFi shops don't use them. Sad

Pages

Log in or register to post comments