Concerning Helvidius's (1) mention of Jesus' "brothers", St. Jerome explains that this designation in Scriptures often signifies a relationship based on "friendship". For example in Psalm 132:1, and in the New Testament where sometimes all Christians are called "brothers" (Jn 20:17; 1 Cor 5:11); or this designation indicates those who have another degree of kinship, as in Gen 27:46; 29:1-12; 31:17.
Jerome rejected the idea suggested in the Protoevangelium of James, that the "brothers" were the sons of a first marriage of Joseph. He explained that in the case of the Lord's brothers they were actually cousins (sons of maternal uncles and aunts). Jerome rested his argument on the fact that philologically speaking, Hebrew and Aramaic do not have a specific word for "cousins" and instead use "brothers" for this designation (e.g. in Gen 13:8; 14:14; Lev 10:4; 1 Chr 23:22).
Simeon was a cousin of the Lord: Jerome knew, even though he did not quote it, the account of Eusebius (Hegesippus’ Memoirs, ca. 180) where it is written that Simeon is a "son of Cleophas, an uncle of the Lord".
James the Less was also the Lord's cousin: son of Mary (Mk 15:40; Mt 27:56) the wife of Alpheus (cf. Mt 10:3; Acts 1:13) and the sister of Mary, the Mother of the Lord.
Jerome also made this comment to Helvidius:
"You affirm that Mary did not remain a virgin. On the contrary, I will go even further and affirm that Joseph, following the example of Mary, also lived as a virgin, so that the virginal Son was begotten by a virginal marriage. In other words, if a holy man cannot be suspected of an extramarital relationship, and if it is not written that he had another wife, if finally he was for Mary, who in the opinion of the people was considered his wife, more a protector than a spouse, then it only remains to conclude that the man who is called the father of the Lord, lived virginally with Mary."
Marian Encyclopedia
(1) Helvidius was the author of a work written prior to 383 against the belief in the perpetual virginity of Mary.