There are two separate vulnerabilities to consider: the risk of being infected, and the risk of developing severe illness. J-pouchers (with or without IBD) probably have identical risk of being infected as the general population. This risk goes up with every human contact, and multiplies as the number of new daily cases multiply, but it’s not likely to be any different for J-pouchers. I’ve chosen to be very, very careful to avoid being infected: I simply won’t go indoors with anyone outside my household, including grocery stores, restaurants, etc., and I’m very careful outdoors, too.
The risk of developing *severe* COVID-19 (once you are infected) is trickier. As @RossG points out, that seems to be due to an immune system overreaction called a cytokine storm. In *severe* COVID-19 dexamethasone (a potent cousin of prednisone) seems to tamp down that overreaction and help manage the severity, and tocilizumab probably works analogously. Unfortunately these drugs (at least dexamethasone) seem to do more harm than good early in a COVID-19 infection (before it has become severe), so being on them continuously when you happen to catch COVID-19 is probably is a net negative.
IBD is sometimes mischaracterized as a hyperactive immune system, and sometimes mischaracterized as an immunocompromised state. While either of those may sometimes be true, IBD is more of a *specific* inappropriate immune response to something that properly belongs in your gut - perhaps a bacterium, perhaps a component of the gut wall. Suppressing the immune system broadly can help control this, which is why prednisone helps control IBD, but we only do that because we have no idea how to suppress just the inappropriate immune response.
I think everyone needs to do a better job preventing the spread of COVID-19. When we take risks in this instance we are taking both personal risk and creating risk in our community. The right public health question to ask before entering that indoor space is “how many people am I willing to kill to engage in this activity?”