I would imagine any additional issues with prostate removal for you would be related to the degree of adhesions (scar tissue) that has formed because of your rectal dissection and j-pouch.
If your urologist thinks this is a big deal, you probably should seek a second opinion from someone with more experience. But, if he is confident in his skills and only being up front about possible pitfalls, that is different.
There are a few here who have had prostatectomy. Have you checked the Men’s Health Forum?
Jan