If it makes you feel better, I am the only person in my family with UC--and this is going back several generations. My brother had a fistula of unknown etiology, but he's had no other problems and has had several scopes which do not show any form of IBD. My sister has mild IBS.
I recall my former GI once saying that the familial/genetic association with IBD is not concrete or even entirely understood, in that one family member having the disease does not necessarily predispose others. He acknowledged that there is a genetic association, more so with Crohns, but he did say that only about 30% of those diagnosed with IBD also have a family member with it, which means that 70% of cases don't. Clearly the 70% weighs much heavier.
I agree with Jan. It's best not to worry about what could happen. We are ALL at risk, all the time, for something or other, even despite healthy lifestyles and educating ourselves about risk factors. And, unlike some other diseases which can be reduced or prevented by adopting a healthier lifestyle, no one has ever proven that IBD can be avoided by changing diet, taking supplements, etc (if only!). So easier to cross bridges when we get to them.