The half life of Imodium is about 9-15 hours. That means that only half of the drug is gone after that time. So, yes, there is some in your system for days. If you are not used to taking it, you should be careful about taking higher doses.
Plus, the fact that you normally do not take it, you probably have a much slower gut normally than any of us who do take Imodium regularly. If you had a quick bout of indigestion or mild food poisoning, the cause could have resolved faster than you metabolize the Imodium.
I would not worry about it, other than to use caution in the future. We are all different, and your own response matters more than how any of us respond to it.