think you're now into the realm of talking about SAMPLE RATE CONVERSION ... You will lose some quality in the conversion

Don't forget that, in this case, the sample rate conversion has already happened once. CDs are encoded at 44.1kHz and having lame encode them at 32kHz means that lame performed it already. And I don't imagine that it has a very advanced algorithm for this. Changing back will put artifacts on top of artifacts.