You're probably not going to harm anything, but the reason that Windows wants you to tell it that you're going to remove storage media is because it doesn't (necessarily) write everything to to storage media right when you tell it to. It often caches it up to be written in bigger, more efficient chunks. If you just yank it out, it might not have the data on it that you just wrote to it. Even worse, it might end up in a weird state where files, possibly even files you didn't touch, might get corrupted.

So, yeah, you should go through that dance.