As just one of many instances of Brigham Young (and other early LDS leaders) explicitly rejecting their infallibility, Brigham once said the following:
I will say a few words in regard to your belief in being led, guided, and directed by one man. Brother Jackman has said that our enemies hate the fact of our being led by one man. Thousands of times my soul has been lifted to God the Father, in the name of Jesus, to make that verily, true in every sense of the word, that we may be led by the man Jesus Christ, through Joseph Smith the Prophet. You may inquire how we are to know that we are so led. I refer you to the exhortation you have heard so frequently from me. Do not be deceived, any of you; if you are deceived, it is because you deceive yourselves. You may know whether you are led right or wrong, as well as you know the way home; for every principle God has revealed carries its own convictions of its truth to the human mind, and there is no calling of God to man on earth but what brings with it the evidences of its authenticity. Let us take a course that leads to the perpetuity of the natural life which God has given us, and honour it. Should we pursue this course faithfully, and never bestow one thought for the life that is to come, we are just as sure of that immortal life as we are of the life we now possess. This, in fact, is the only way in which we can be prepared to inherit that more glorious life. What a pity it would be if we were led by one man to utter destruction! Are you afraid of this? I am more afraid that this people have so much confidence in their leaders that they will not inquire for themselves of God whether they are led by Him. I am fearful they settle down in a state of blind self-security, trusting their eternal destiny in the hands of their leaders with a reckless confidence that in itself would thwart the purposes of God in their salvation, and weaken that influence they could give to their leaders, did they know for themselves, by the revelations of Jesus, that they are led in the right way. Let every man and woman know, by the whispering of the Spirit of God to themselves, whether their leaders are walking in the path the Lord dictates, or not. This has been my exhortation continually. (Brigham Young, January 12, 1862, JOD 9:149-50; emphasis added)
Just another passage from the Journal of Discourses critics will gloss over . . .