Just spitballing here. What about a switch with two positions that allows you to switch between the two battery banks? In Position A, your two 6v batteries would be connected to the system. In Position B, your 12v deep cycle would be connected to the system. Something like this maybe:
https://www.amazon.com/Blue-Sea-Syst.../dp/B0070YEEI8
You'd have a single ground to the system, but your positive would switch back and forth between your 12v and your 2x6v batteries. That would ensure that you don't have both "battery systems" connected to the system at the same time. Plus, that particular switch is a 4-position switch that allows you to connect both banks at the same time. Might give you some flexibility in the future. Otherwise, there's a 3 position switch in that same entry.
Just curious, is there a reason you don't have your 12v fridge pulling power from your 2x6v battery bank?