Are you sure that you want to attach the flag pole to the ladder? If it get windy that could add a lot of stress on your ladder supports. Also, if there is a bed in that area at night you may be kept awake by the vibrations transmitted to the ladder by the flag pole.... may not let you get a sound sleep.
If you still want to do it, why not get a length of 1.5" PVC, an end cap and put a couple of "I" bolts through it for the flag and attach the PVC to the ladder.
Another option is a telescoping flag pole like the one at Harbor Freight:
Flag Pole Kit - 20 Ft. Telescoping Flag Pole
You could mount it on the back bumper and attach it to the ladder. Easy to set up and take down.
Or, how about the spinning PVC flag pole?
Rotating PVC Flagpole Flag Pole Plans - Instructions - Embroidered Flag
Just my thoughts,
Don