This armor is rare because of how strong it is. You can find it rarely off of defeated enemies or inside item containers. You will need to be a high level to get the grade "X" armor.
If you complete "Pinnacle Station" DLC mission, there is a chance you will get QUARIAN Colossus X armor every time you pay the 222,000 credit shipment in the apartment on the planet Intai'ei. You can also get HUMAN MEDIUM Colossus X armor this way.
If you complete "Bringing Down the Sky" DLC mission, you can gain HEAVY Colossus X armor if you ask for "Heavy Armor" as a reward for finishing the mission. You'll need to be a high level to be given grade "X" armor.