Regardless of you having a pristine pouch, the fact that you have serious perianal disease is a big deal. It may not be an indicator of Crohn’s because perianal disease that arises from the suture line at the anus could a surgical complication, not IBD related. Finding the opening of a fistula can be quite challenging, so maybe it is time for a second opinion from another provider with more experience.
If this was pouchitis or cuffitis there should be some visual evidence when scoped, even if you were on flagyl.
In the meantime, it might make some sense to try to resume Remicade or try Humira to see if it might help close the fistula.
Jan