Is there another contractor who works with the state? Maybe you can change your agency but keep your job?

Part of my contract is that I can't work for the same employer directly after leaving the contracting agency without paying them some absurd amount of money. Even so, I've been considering it.