I used to take mine out for the winter, but the new unit we bought is still at the dealership and they leave them in for the winter.
I guess that shows that it won't hurt them at all. The only reason to take it out would be for concerns of theft, as already pointed out.
2010 Jayco Eagle 351RLSA
2011 Nissan Titan 4x4 (don't worry, I don't pull the Eagle with it!)
Currently shopping for a Silverado 3500 Duramax
Oh, and I have a tent... Somewhere around here....