The Shulhan Aruch (Orah Haim 124:4) says that when the Shaliah Tzibur recites the repetition of the Amida, there should be at least nine congregants who listen and respond to the Hazara intently. If there are not nine who can respond, the Shulhan Aruch continues, there is a possible risk of the Shaliah Tzibur’s blessings being recited in vain. As such, each congregant should consider it as though he is the ninth person and should therefore respond to the blessings.
There are situations in which having nine people responding is not readily available. For example, if there is a Minyan of exactly ten men and a few pray especially slowly, then it could be burdensome to wait for nine to respond. Or, there are congregations in which the congregants simply do not have the requisite level of concentration to respond during Hazarat Hashatz. In such circumstances, Rabbi Baruch Toledano (Kitzur Shulhan Aruch, ch. 111, § 15 and introduction to Hashamayim Hadashim) says that the Shaliah Tzibur may recite the Hazara but should have in mind that the recital should have the status of a voluntary prayer. He also points out that this opinion is cited by the Mishna Berura (M.B., O.H. 124:19) in the name of the Shulhan Shlomo, as well as by Rabbi Shlomo HaKohen of Debdou, Morocco.
It should be noted that this approach does not work on Shabbat or Yom Tov, during which voluntary prayers are not recited, but rather only on weekdays.
Summary: At least nine men should listen and respond to the Hazarat Hashatz. If this is not possible, the Shaliah Tzibur should have in mind that the Hazara have the status of a voluntary prayer.