I've actually been there (Okura) once with my female friend and its true that once they see an actual girl (and not a newhalf/ladyboy), they will swarm like flies. However, the men surrounding you are pretty respectful with boundaries and won't do anything especially if you just wave your hand...