Each player chooses a color. Then exile each permanent unless it's colorless or it's only the color its controller chose.