are you on ps3? I'm xbox and I play a lot.
I have also noticed the gtag crew doesn't join up. I think it's the way invites are sent. you basically get a random phone invite and people typically only join if their friend invites them, otherwise you dont want to play with randoms.