Same for me. I live amoungst houses with walls, and white cars idling outside them at all hours. It’s probably the safest place to live in Tokyo. Even while all of Tokyo is safe. The home life does not bring their work with them.
Even in a local pub. There are lower level management, but not hoods, that will take sporting bets, without hustling~
Years ago when I lived briefly in Japan I met quite a few. Back then the Yakuza were not an illegal organization (yes I know I just dated myself) and they actually had offices that you could go to. I've also had occasion to meet a few outside Japan on different jobs. I've found that in general they are very laid back guys with good attitudes. When they group together they tend to get a serious herd mentality going where they take affront to many things they would ignore individually.
I did when I went to Tokyo for my first time. The guy seemed pretty pissed that 3 Americans were eating in his establishment, I was the only one who noticed and tried to get the group out of there, but alas they are pretty hard to handle, much like children. It took me saying check and paying for everyone's meal before we left. Once outside they wanted to know why we left and they were enjoying their time and drinks, I told them and their jaws hit the floor as they couldn't believe what I had said.
To be honest I was a bit scared to ask at the time. He ran a bar and a lot of them would come and do money runs etc. he had a full body suit of tattoos going on. Oh and they always went to onsen and Sento together lol