Should be possible to support but the Darwin xorg-server build assumes it is going to build xquartz and nothing else.