Sorry to hear of your results. I am 20 years post takedown and really have not had the same issues as you. I have a pattern of mild inflammation/pouchitis below the inlet and above the outlet which has remained in a "simmering" state for about 17 years with antibiotic treatment. I would probably have the same concerns as you, but perhaps you will respond well to the rectal meds and the next scope will show that. Our GI has also told me that the inflammation in the rectal cuff could be microscopic and lead to the same concerns as far as cancer of the rectal cuff. So basically all of us, not just you, have the same concern which needs to be addressed with regular (annual) scopes and biopsies of that tissue.
The fact that you are somewhat asymptomatic is likely a good sign, and if symptoms return there is always antibiotics - which has controlled my situation for 17 years with no evidence of causing other issues, possibly because I have rotated them every 3-4 weeks.