It is important to respect the choice of students who do not want to take part. Try to make sure these students feel comfortable about their choice. One way is by using language that normalizes their decision not to participate.
"Sure, that’s fine to leave the class if you’d prefer. We’ve set up the library for anyone who doesn’t want to participate. We’re not always up to a big discussion—it’s important to have choices."
Not everyone is ready to share her or his opinions, feelings or experiences; choosing not to participate may be a way of protecting themselves.