They quarried the stone but it was Scottish masons who dressed and laid the stone, not slaves.
Current sources take the politically correct narrative and say they did help build the White House but they can only point to labor in the quarry.
https://www.nytimes.com/2016/07/27/us/politics/michelle-obama-white-house-slavery.html?_r=0