Joseph also remained a virgin, out of consideration for Mary

You say that Mary did not remain a virgin. I, on the contrary, wish to affirm something more radical—that Joseph actually remained a virgin—and did so for Mary, so that a virgin Son could be born of a virgin marriage. Since in a holy man like Joseph there was no room for fornication, and since it is not written that he had another wife, and if he was for Mary, who was considered his wife, a guardian more than a husband, we can conclude that, together with Mary, the one who deserved to be called father by the Lord also remained a virgin.

Saint Jerome