I have both, now. If I didn't have a fistula and need a seton, etc, I'd probably only see my GI. My current GI only works with IBD patients, and has a lot of pouch patients.
If you no longer have surgical issues, a medical GI with good knowledge is likely all you'd need, unless you run into issues. Most pouch issues down the road are not surgical issues.