I don't worry about it. If I have to do it, I do it. I am not sure anyone can tell you if it damages something or not, just thoughts/opinions and they will be at both ends of the spectrum.
It's a good practice to look over your tires, wheels and suspension components from time to time. This should show any issues before you have a big issue.
2016 Jayco Eagle HT 29.5BHDS (ordered 12/30/15, delivered 3/8/16)
2015 F-350 crew cab, short bed, 6.7L PSD, Pullrite Superglide 3300 hitch