I'd recommend asking your GI doc - or your colorectal surgeon - or your GP - to order you a pelvic MRI. Scopes do not always catch the fistula hole, it can be very small and with all the folds in there it can be somewhat hidden. GIs generally pass fistula patients on to CR surgeons, in my experience.
What you are describing does sound like a fistula, but an MRI can give you a better idea. I don't know anything about cultures or granulation tissue, etc. (sorry)...
I hear you on the concern of sounding like a hypochondriac - I had symptoms off and on for years - asked my surgeon if it could be a fistula and he LAUGHED at me and said if I had a fistula he would be able to see it. SO not true, especially for an RV fistula. I finally got my GP to order the pelvic MRI after I developed an abscess and was a total mess; the fistula was finally found, I went to Cleveland and now have a seton.
You should not let this go - you want to know what it is and explore options to treat it if it is indeed a fistula.
Let us know what's next for you...