See if your BIOS allows specifying a boot-device search order. "offboard" or "SCSI" should be at the tail end of the list, or not on it at all.

If you're running Linux, then you can probably remove the BIOS chip from the expansion card and eliminate the problem that way.