It would be better I think, but I still think my original idea is more useful for driving situations where you can think in your head what letter you are on

For example, you could do a button press for 2 seconds could mean next letter, and then a quick click just means next playlist entry. That would be the best situation I think.