I bought a utility brush, kind of like a regular toilet brush but round with no bend. I tried scrubbing the stuff off from below the flapper and couldn't get much of it off. Sticks pretty tight.
I don't look too closely any more if I can help it. I guess it's not hurting anything, other than my sense of pride when I have guests over.