There usually is a slight performance penalty for not having the same size and type of memory on both channels. The penalty is so small however that you're not going to ever notice it unless you start running memory benchmarking utilities.