If you're using Homebrew for installing packages (like many does), it seems as if Homebrew won't register that you have the CommandLineTools installed if you move the/Library/Developer/CommandLineToolsdirectory, and will try to compile some bottles from source.
Homebrew apparently needs some tools found in the separate Command Line Tools package installed with sudo xcode-select --install that's missing from the regular Xcode install (I noticed this when upgrading to a new version of Python, it was still being compiled from source no matter what I tried.)
So, if you use Homebrew, you're probably better off keeping the Command Line Tools installed at the default location.Just install as described with the sudo xcode-select --install command and leave it there, it's not taking up much space anyway.