Why are some books in the Bible while others were excluded and how was that decided?
This is a great question! Today I’m going to address the New Testament canon. Let’s start off with a little bit of context.
By the end of the first century or early in the second century, all of the texts that now make up the New Testament had been written. As early as AD 95 or 96, Clement of Rome alludes to multiple passages in the New Testament, though he does not name them as scripture yet (that is a title still reserved for the Old Testament at this point). In the mid-second century, St. Justin Martyr refers to the gospels as the “memoirs of the apostles” in his First Apology, and in the famous text Against Heresies of St. Irenaeus (c. 180), the saint firmly proclaims the authority of the four gospels.
At the same time, a number of other texts were circulated amongst the nascent Christian community. There are really two categories of texts outside of the canon of Scripture. First, texts whose teachings were accepted in whole or in part as edifying for the Christian community but ultimately deemed outside of the core texts which became the New Testament, and second, texts that were considered heretical.
Concerning the first category of texts, in the first and second centuries, many these early Christian writings were considered interchangeable with the four Gospels, the epistles of Paul, and the catholic epistles. For example, St. Clement of Alexandria considered books such as the Didache and I Clement as part of the canonical list of texts while later in the early fourth century, Eusebius the historian points out that James, Jude, 2 Peter, 2-3 John, and Revelation were sometimes disputed as parts of the New Testament.
Eventually which books were accepted and which were considered outside of the canon but still useful to Christians sorted itself out. The first full list of the current 27 books of the New Testament that we have is in a Paschal letter of St. Athanasios in 367. However, the Church continued to quote and learn from the letters of Clement, Ignatius, Polycarp, and Barnabas as well as the Didache and the Shepherd of Hermas. These works later became known as the works of the Apostolic Fathers, those who were typically one or two generations away from the apostles themselves and shared much of their worldview and theological emphasis.
The second group of texts that were being written and circulating at the same time as the New Testament and the writings of the Apostolic Fathers were typically texts associated with gnosticism. Gnosticism was not a single heresy but a group of heresies identified generally by the rejection of the material world as evil and created by a lesser, created god (called a demiurge) and salvation as “enlightenment” or freedom from this completely corrupted or even simulated realm. Practices and beliefs amongst gnostics varied widely.
Where things get tricky in the modern mind is that many of the texts associated with gnosticism were attributed to the apostles and used Christian terminology such as “gospel” to describe themselves, and even sometimes quoted Christian texts as justification for their clearly non-Christian beliefs. Unlike the canonical texts, however, most of the gnostic texts were written in the 2nd-4th century after the established circulation of the New Testament gospels. These include such titles as the Gospel of Judas, the Gospel of Mary, the Acts of Peter, and many others. Unlike some of the Apostolic Fathers, the gnostic texts were rarely included in early lists of the New Testament and were most usually blatantly marked as heretical (for example, by Irenaeus and Eusebius).
Removed from the living Tradition of the Church, the modern era has resurfaced many of these gnostic texts and has presented them as if they are newly-discovered and were kept hidden by early Christian leaders for some nefarious reasons. From inside the ancient Church, we can easily see that line of reasoning for what it is: nonsense. The Church never accepted gnostic teaching or the texts associated with it, and the reason that these texts disappear or become rare and obsolete is because they were not important to the actual practicing Christian community. All one has to do is read a few passages from one of these books to see that the Church very easily discerned their heretical teaching and vastly different worldview.
Thus, those books which were apostolic in origin and central to to the message of salvation were selected to be the main texts of the Christian faith while others were identified as useful to Christians, but not central to life of the Church or as simply heretical. All this is to say that the texts of the New Testament were written, collected, and canonized over the course of the first four hundred years of Christianity, and that ultimately, it was the consensus of the whole Church, voiced through the Fathers and guided by the Holy Spirit, that ultimately determined the canon of the New Testament.