Commenting on the claim that the “unanimous consent” of the Fathers supported the interpretation that the “rock” in Matt 16:18, Yves Congar, a leading Catholic theologian, wrote the following admission:
Unanimous patristic consent as a reliable locus theologicus is classical in Catholic theology; it has often been declared such by the magisterium and its value in scriptural interpretation has been especially stressed.
Application of the principle is difficult, at least at a certain level. In regard to individual texts of Scripture total patristic consensus is rare. In fact, a complete consensus is unnecessary: quite often, that which is appealed to as sufficient for dogmatic points does not go beyond what is encountered in the interpretation of many texts. But it does sometimes happen that some Fathers understood a passage in a way which does not agree with later Church teaching. One example: the interpretation of Peter’s confession in Matthew 16:16-19. Except at Rome, this passage was not applied by the Fathers to the papal primacy; they worked out an exegesis at the level of their own ecclesiological thought, more anthropological and spiritual than juridical. (Yves M.J. Congar, Tradition and Traditions: An historical and theological essay [London: Burns & Oates, 1966], 398-99)
For an excellent book-length refutation of the dogmatic teachings of Catholicism on the Papacy in light of the patristics, see Edward Denny, Papalism : A treatise on the claims of the papacy as set forth in the encyclical Satis Cognitum (1912)