Since Joseph was just and unwilling to expose Mary, why had he a mind to put her away ? I give you on this point not my own opinion, but that of the Fathers. Joseph’s reason was the same as Peter’s when he said, “Depart from me, for I am a sinful man, O Lord”; and that of the centurion when he exclaimed, “I am not worthy that thou shouldst enter under my roof.”
Joseph looked on himself as a sinner and as unworthy to entertain one in whom he beheld a superhuman dignity. He beheld with awe in the Virgin-Mother a certain sign of the Divine Presence, and as he could not penetrate the mystery, he wished to put her away.
Peter was struck with awe at the greatness of Christ's power; the centurion by the majesty of His presence.
Saint Bernard, excerpt from the 2nd sermon “Super Missus est”