First off, as you already know you're not ruining anyone's life. I hate to take anyone's side without knowing all the details and actually being a part of their lives, so I'll say this: we all can say some harsh stuff when we're heated in the moment. Some people are just better at not letting those things slip.
When I was diagnosed I was abandoned by loved ones in a way. I say "in a way" because they really didn't support me at all. They didn't know what this disease was and how bad it can get, and they treated my like someone who just had a tummy ache. When i realized they weren't giving this disease any weight, I pushed them away because it was pointless to have someone around that treats you like you are faking it or making it bigger than it is. Now here we are 8 years and a few surgeries later and they have nothing but respect for me.
If you feel that this person really loves you and just slipped up with their word choice, then absolutely move forward and don't let it weigh you down. This disease is hard on the loved ones around us in ways that sometimes we don't understand. At the end of the day only you know what's best for you. If he's making you feel bad about your disease often that's gonna wear you out quickly and you don't need that. You've got enough to battle as it is.
Curious to hear an update on what's going on! Sending you love from California!
David