This must be very stressful for you, to feel so much is weighing on your decision. I had two kids with a J pouch with no problems (I went to an OB who had experience with higher risk pregnancies, and did a scheduled C-section to avoid any problems with vaginal delivery). I had the pouch already for about 10 years by the time I had my first child. Is your doctor talking about infertility due to the J-pouch? or due to the disease? Are you somewhere where you could get a second opinion?
Make sure you take care of yourself, it takes a while to recover from surgery. Tell yourself you don't have to decide right away and give yourself time to do some more research while you are recovering.
Sending you some support from afar.