yes, you need to get to a doctor sooner than later. A lot of people, myself included, do not get fevers and/or flu like symptoms with abscesses.
Have you seen Dr. Remzi before? If you have, you can try contacting his nurse Vicki and see also what she advises.