Are you sure that you want to attach the flag pole to the ladder? If it get windy that could add a lot of stress on your ladder supports. Also, if there is a bed in that area at night you may be kept awake by the vibrations transmitted to the ladder by the flag pole.... may not let you get a sound sleep.
If you still want to do it, why not get a length of 1.5" PVC, an end cap and put a couple of "I" bolts through it for the flag and attach the PVC to the ladder.
Another option is a telescoping flag pole like the one at Harbor Freight:
Flag Pole Kit - 20 Ft. Telescoping Flag Pole
You could mount it on the back bumper and attach it to the ladder. Easy to set up and take down.
Or, how about the spinning PVC flag pole?
Rotating PVC Flagpole Flag Pole Plans - Instructions - Embroidered Flag
Just my thoughts,
2013 Jayco Eagle 284BHS
250Watt Grape Solar Panel, MorningStar MPPT 60 Charge Controller
1500 Watt Ramsond PSI, 2 Trojan T145 Batteries (260Ah)
2 - AirSight Wireless IP Cameras (used as rear view cameras)
EnGenius WI-FI extender, D-Link wireless (n) modem
MagicJack Internet Phone
2012 Ford F150XLT, EcoBoost w/3.73
157" Wheel base, HD Towing Package
Our Solar Album http://www.jaycoowners.com/album.php?albumid=329