Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Topic Options
#21276 - 28/10/2000 21:34 detailed analysis of mp3 encoders
borislav
addict

Registered: 30/04/2000
Posts: 420
Loc: Sunnyvale, CA, USA
Probably many of you have seen this already, since it appeared on Slashdot today, but if you haven't, it's definitely worth checking out.

http://www.airwindows.com/encoders/index.html

It's a comparison of various versions of LAME, BladeEnc, and Fraunhofer encoders at different bitrates with detailed analysis of frequency response, pre-echo and over-ring characteristics.

Borislav


Top
#21277 - 28/10/2000 23:01 Re: detailed analysis of mp3 encoders [Re: borislav]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
Yeah, it's too bad the texture they used for that page background makes it physically painful to read through the text.

___________
Tony Fabris
_________________________
Tony Fabris

Top
#21278 - 31/10/2000 11:16 Re: detailed analysis of mp3 encoders [Re: borislav]
dewdman42
member

Registered: 13/09/2000
Posts: 186
Very interesting read actually. Now I'm even more confused then ever about which encoder I should use though. ;-)


Top
#21279 - 31/10/2000 11:50 Re: detailed analysis of mp3 encoders [Re: borislav]
Dylan
addict

Registered: 23/09/2000
Posts: 498
Loc: Virginia, USA
It is an interesting read as is r3mix. My problem with both of these sites is that they focus almost exclusively on objective, measured analysis of manufactured source material. While there is merit in that sort of evaluation there is also merit in subjective evaluation of real music. I would like to read an analysis that balances somewhere between what is presented in these sites and the subjectivist drivel in magazines like Stereophile.

I've done some listening tests of my own but they were far too uncontrolled for me to present them to anyone else as anything but narrative. I've considered trying to set up something more controlled and with a larger sample group than just myself, but I've yet to muster the motivation

-Dylan



Top
#21280 - 31/10/2000 12:29 Re: detailed analysis of mp3 encoders [Re: Dylan]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
Excellent, excellent point Dylan.

In fact, I just got done discussing this very subject with Tanstaafl in private. Although the objective measurements are vital to knowing the details of how the encoders perform, what good does it do if there's no audible difference from one encoder to the next?

My problem with the R3mix tests and this other recent test is this: The tests were made by computer hackers, not audiophiles. What we need, in addition to those detailed tests, are double-blind listening tests made by audiophiles with golden ears, using samples of real music.

I actually saw such a test, once. Unfortunately, it was made over a year ago, using older versions of a few different encoders. Since the encoders have changed so much with each version, the results of that test are useless today.

The test was done by a music magazine, and the testers were their audio reviewers-people whose jobs depended on being able to hear subtle differences in sound reproduction. The test had a very interesting protocol, one which I think is possibly the perfect protocol for tests of this type.

The author of the test first chose small musical passages from a variety of styles. There was classical, single dry solo vocal (very good for hearing certain kinds of artifacts), rock/pop (including, to my pleasant surprise, the filter-sweep passage from "Ray of Light"), and some others.

Then he ripped those sections into .WAV files and encoded them to MP3 with the encoders he was testing. Then he decoded the resulting MP3 files back into .WAVs and burned those to a CD as audio tracks in a very interesting way:

Each "test" was a group of tracks. It would start with the original source file as the first track. Following that track would be the encoded versions scrambled randomly with a second copy of the original source file again. So unless you really had golden ears, you wouldn't know whether you were listening to a copy of the original track or an MP3-encoded one.

The listeners' job was to rate each of the "following" tracks in comparison to the first track in the group (the known original). The "control" was the repeat of the original track hidden among the encoded tracks- in theory, if the listeners were doing their job right, the original track would rate as perfect and the encoded versions would rate as somewhat less than perfect. This was the "blind" part of the experiment.

What made the experiment double-blind was that the author of the test made a few different sets of the CDs with the tracks randomized in different orders. Each one was given a serial number, and the actual order of tracks was recorded according to the serial number. But which listener got which CD was randomized, and not known to the test author until the end when the results were tallied.

In essence, it was a perfect double-blind listening test. It tested only the encoders' sound quality, not the quality of the playback equipment. Each listener got to listen on their favorite home equipment using their favorite CD player. An exact, immediate, side-by-side comparison of the encoded versions to the original was possible, but without the "confirmation bias" of knowing whether they were really listening to an encoded version or not.

Even with the strict protocol, it was amazing how well these audiophiles were able to pick out the differences among the encoders. They consistently rated the various encoders lower than the "control" track and noted their reasons for doing so. Out of the whole test, there was only one abberant "spike" of one tester mistaking the control track for an encoded track. I was impressed.

Now I want to see this done for recent versions of LAME vs. Xing at high bit rates.

___________
Tony Fabris
_________________________
Tony Fabris

Top
#21281 - 31/10/2000 18:31 Re: detailed analysis of mp3 encoders [Re: Dylan]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
I've done some listening tests of my own but they were far too uncontrolled for me to present them to anyone else as anything but narrative.

Well, Dylan, please narrate! Reading your posts on this bbs, it is clear to me that you are well informed and can express yourself well. I, for one, would be most interested in hearing what you have to say!

tanstaafl.

"There Ain't No Such Thing As A Free Lunch"
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#21282 - 31/10/2000 20:41 Re: detailed analysis of mp3 encoders [Re: tfabris]
Terminator
old hand

Registered: 12/01/2000
Posts: 1079
Loc: Dallas, TX
I have read that article as well. I remember that wma was given one of the lowest ratings at that time. Actually I think the author said different encoders were better at different types of music. Maybe you convince the magazine to do another test with newer encoders?

Sean

Empeg 12 gig green 080000078

Top
#21283 - 01/11/2000 11:26 Re: detailed analysis of mp3 encoders [Re: Dylan]
mcgrant
journeyman

Registered: 28/04/2000
Posts: 84
Loc: Menlo Park, CA USA
It is an interesting read as is r3mix. My problem with both of these sites is that they focus almost exclusively on objective, measured analysis of manufactured source material.

Well, it really depends on how you look at it. It's certainly true that they spend a lot of time on charts and frequency plots. But in r3mix's case, they have one important reason for it. r3mix's plots show that many MP3 encoders indiscriminantely remove high-frequency information in order to increase compression. The frequency/sensitivity curve of the human ear just isn't that abrupt. So when encoders do that, they will inevitably sacrifice audible sound quality.

r3mix also refers to a controlled listening test that suggests that a good encoder with 256kbps CBR cannot be distinguished from the source on a statistically consistent basis. Unfortuntaely the source they refer to is in German so I can't read it too well...



Michael Grant
12GB Green
080000266
_________________________
Michael Grant 12GB Green 080000266

Top
#21284 - 02/11/2000 01:30 Re: detailed analysis of mp3 encoders [Re: tanstaafl.]
dewdman42
member

Registered: 13/09/2000
Posts: 186
Wow...what a cool test with the CD's and all. I would LOVE to see something like that take place again with the latest encoders.....

-steve


Top
#21285 - 04/11/2000 11:02 Re: detailed analysis of mp3 encoders [Re: mcgrant]
smu
old hand

Registered: 30/07/2000
Posts: 879
Loc: Germany (Ruhrgebiet)
r3mix also refers to a controlled listening test that suggests that a good encoder with 256kbps CBR cannot be distinguished from the source on a statistically consistent basis. Unfortuntaely the source they refer to is in German so I can't read it too well...

I think you refer to the article by the german c't magazine that can be found at http://www.heise.de/ct/00/06/092/default.shtml. I just sent an email to the original author of that article asking him for permission to translate the article and publish it on the web (I proposed to publish it on the magazines homepage along with the original article). Already having done this before for an other article, I expect to get that permission.


cu,
sven
(MkII 12GB blue, #080000113)
_________________________
proud owner of MkII 40GB & MkIIa 60GB both lit by God and HiJacked by Lord

Top