Hi there! I had this exact situation. No problems at all with my j-pouch since it was constructed in 2004, but last year I had a huge ovarian cyst that wouldn't go away after several months of surveillance. I'm at an increased risk for ovarian cancer, so my doctors felt that it needed to be looked at/removed. I had several conversations with my gyn-onc about my desire to have a colorectal surgeon present or at least on call during the surgery. She seemed to feel confident that she could handle it herself but told me that she would talk to one (not sure if this actually happened). Long story short, the surgery did end up being quite a bit more complicated due to scar tissue and adhesions (from bowel surgeries) surrounding my ovaries. She ended up having to remove the entire ovary, but I think that was more because the cyst was so big rather than a result of my previous surgeries. (Side note/tip so you can learn from my mistake- My recovery from the surgery was harder than I expected because it took my bowels almost a week to "wake-up," and I had awful gas pain/couldn't keep anything down in that time. I had kind of ignored their advice to take colace/metamucil ahead of the surgery because I thought I really didn't need any help in that department, but in retrospect I wish I had listened to that part :-))
I will be having my other ovary removed in the next 10 years to reduce my cancer risk, and my surgeon commented that when it comes to that surgery, she will definitely want a colorectal surgeon involved because she saw that some of my bowel is attached to that one, and she's not sure she will be able to identify what is ovary vs. bowel. I'm pretty sure I'll be seeking out a major research hospital for that surgery since I've lost some confidence in my current doctor as a result of this experience. (I live in central VA and this surgery was done at UVA, so I may be looking at the DC area in the future.)
SO all that to say, I think you're wise to be considering where you want this surgery performed. If I were you, I would probably be willing to travel to get to a place with the most experience and expertise. Best of luck, and let me know if you have any other questions!