Without knowing a lot more about how the modem deals with its traffic there's no way to know for sure. However, given that you have a computer of some nature behind the modem and still communicating with something on the other side of the modem via Ethernet, it seems to me that it is possible for such data to leak out under the right circumstances. Effectively, anything plugged into that switch, even on the other side of your broadband link, is going to be on the same physical Ethernet (layer 2) network, and anything that can gain access to that network has an avenue of attack. If you could segregate those extra ports into a separate VLAN then you should be okay, but even if the switch supports VLANs, which it probably doesn't, you probably don't have enough access to configure it.

My guess is that if you're paranoid enough to ask, then it's probably not safe enough for you.