It's possible that in your first case all the settings were fine, except for some reason the VBR header didn't get written to the file. The symptoms you describe sound a lot like what WinAmp does on a VBR file that doesn't have the appropriate header. I'd use some tool like VBR fix to fix the header, and then see what WinAmp says when you play it.