Depends on the frequency range used by the subwoofer. Yes, any time alignment change here will have a tiny impact, but it shouldn't be noticeable until the time alignment value is large enough to impinge upon the woofer frequencies.

If your SW, for example, cuts out for anything above 100Hz, then any time alignment value below 10ms is unlikely to be a problem.

Cheers