I'm more likely to not invite someone back for not talking. If someone talks a lot, I can usually shut them up and control them. But with people who don't talk, if they don't really want to talk, they probably shouldn't be on this show, and that's fine. They're talented people with things to say, but sometimes people say what they have to say through other means than arguing.
