Whereas the vast majority of blessings are rabbinic in origin, the obligation to recite Birkat Hamazon has its source in the Torah. Regarding biblical commandments, there is a concept of “Safek DeOraita LeHumra”, that in cases of doubt, one must be stringent in their performance. So too with Birkat Hamazon, one would need to be stringent and repeat its recital if one were unsure if it was recited or not. Additionally, regarding the section of “RetzeVehahalitzenu” that is added on Shabbat, if one recited Birkat Hamazon and is certain that one forgot this addition, one would need to repeat Birkat Hamazon.
There is a discussion, however, if one is unsure as to whether or not “Retze” was recited. Rabbi Shalom Messas (Tevuot Shamesh, Orah Haim, § 39) explains that in such a case, one knows that one recited Birkat Hamazon, which is a biblical obligation but is only in doubt about “Retze“, which is a rabbinic obligation. Following the principle of “Safek Derabanan Lekula”-that in cases of doubt involving rabbinic commandments, one may be lenient-he says that one need not repeat Birkat Hamazon. Other rabbis, including Rabbi Ya’akov Orenstein (Yeshuot Ya’akov, 422:3), side with this position, and explain that since one normally sings special songs for Shabbat during the meal and do other things unique to Shabbat, one most likely made a point of reciting “Retze” in Birkat Hamazon.
Conversely, the Mishna Berura (O.H. 188:16) rules strictly and says that if one was unsure if one recited “Retze“, one would need to repeat Birkat Hamazon. The Mishna Berura’s reasoning is that, since one recites the regular weekday Birkat Hamazon more often, one most likely recited the regular version on Shabbat without “Retze“. In his approbation to Rabbi Messas’ Tevuot Shamesh, Rabbi Ovadia Yosef cites the Besamim Rosh (§ 287), who uses the same logic as the Mishna Berura and rules that one would need to repeat Birkat Hamazon. Nevertheless, Rav Ovadia concludes by saying that this ruling needs further analysis, and elsewhere (Yabia Omer, vol. VII, Orah Haim, § 28) he rules like Rabbi Messas on a similar matter. Rabbi David Yosef (Halacha Berura, Hilchot Birkat Hamazon) also concurs with this opinion.
Summary: If one is unsure if one recited “Retze Vehahalitzenu” on Shabbat, one need not repeat Birkat Hamazon.