We also have a 23b and found we've gotten faster at set up and take down over the years. We load and unload only food, clothing, and bedding after each trip. We use laundry baskets to carry everything from house to 23b and back again. We leave board games, cards, utensils, plates, cups, flashlights, pots/pans, tools, yard games, etc. in the trailer. We put fitted sheets on the fold out tent ends as well as the couch to keep it cooler in the summer, and wash them after each trip. We each have a sleeping bag that we use as blankets. For another seating spot at the table of the 23b, we purchased a small storage bin that is padded so we can sit on it as well as store beach towels and a couple of blankets inside.
As for set up/take down...on the morning we leave, as soon as we wake, we strip down the bed areas, then eat a quick breakfast of cereal and milk. As I clean, the hubby folds the ends in and places the bikes on the back bumper...in addition to taking care of other outside things. We try to do other things the night before like load the folding chairs into the truck after we're done using them. For setting up, I do the inside, he does the outside. However, we also have three older kids who are able to help us and that makes a big difference. As your son gets older, he will be able to help out.
Also, as you do it more often, you will find a strategy that works for you.
2012 - 33 days camping
2013 - 20 days camping
2014 - 15 days camping
2015 - 15 days camping