Even if the enemy is there, the flag should still lower, just not a new one be raised.