A closed mouth doesn't get fed. If they don't want to move (for whatever reason) then they can simply say so. It's not impolite to ask, but it would be impolite and inconsiderate for them not to move down one seat. I think not asking a simple and reasonable thing like that indicates a fear of confrontation. If they don't want to move then they're either rude or have a good reason for it. I say give them a chance to be polite and move over for you.