He married Elizabeth Bouldin, July 3, 1788, daughter of James Bouldin and Sally Watkins, of Charlotte County, Virginia.
James was a large land owner in Lunenburg County and resided there until 1802, when he, with his little family, followed his father and other relatives to Abbeville District, South Carolina. He was a large planter in South Carolina until 1818, when he followed his sons to the territory of Alabama, his older sons having settled in that part of the Mississippi territory called Alabama in 1812. In Alabama, he settled on a large plantation in Madison county, where he lived and died.
In the American Revolution, James Collier was wounded at the battle of Eutaw Springs by a sabre cut across his cheek, in a hand-to-had encounter with a British soldier. He killed the soldier and carried the scar on his face to his grave.