It looks like I might have fixed it. I moved my local wifi network to the top of the "preferred" networks list. There were no networks above it that are visible from here (they are all networks I use elsewhere), but I guess it must have been making it stop and look for those other networks when switching user. Or something like that.