On the back of the album jacket, Rivers wrote that he chose the title "The Brotherhood of Man" because he believed brotherhood is a basic prerequisite to any act of worship in the Jewish and Christian traditions and that the constant message of the prophets was that God reminded us that if we come to the altar and there remember anything that separates us from our brothers, we should go first to be reconciled to our brothers, and only then return to the altar. Music ought to be a bridge of unity and understanding between brothers of different cultures. In the Brotherhood of Man, many cultural elements are incorporated and integrated, including the music of American Black churches, Gregorian chant, jazz, classical chamber orchestration, and contemporary popular music.

The album was recorded in 1967 with Maestro Erich Kunzel conducting the Cincinnati Chamber Ensemble, Chamber Chorus, and the Symphony Jazz Quartet. In addition, The Brotherhood of Man was featured at an Ecumenical Concert in 1968 with the Cincinnati Symphony Orchestra. The album jacket notes that the jazz version of The Brotherhood of Man received its world premiere one month from the recording at the 1967 Newport Jazz Festival. Rivers himself conducted a 50-voice mixed chorus of Boston College Summer Students and Xaverian Brothers, a group which he personally assembled and rehearsed the previous week at Boston College. Accompaniment was provided by the jazz pianist Billy Taylor and the Billy Taylor Trio.