I might have you beat in number of surgeries. My husband told me to have them put in a zipper. I hate wearing that 6" wide elastic belt so really appreciate you sharing your doctor's guidance. It really bothers me as I have fibromyalgia and most bras bother me too. My surgeon didn't tell me to wear it and I do think the nurses fit me for it in order to keep the appliance on longer. I have been wearing it for a few hours after I put on a new bag. It is worth that time because it's reduced the worrying about bag failures and increased appliance wear time.
I switched my care to the Mayo Clinic. My GI there helped me deal with my difficult j-pouch until he and I decided it was time to go. My surgeon, Kellie Mathis there is wonderful. She not only for removed my j-pouch but also diligently worked on my abdominal adhesions. I'm going to ask her directly about using the belt "forever" as I won't be able to do it all the time.