Pickling is done to remove all the contaminants from the surface of the stainless steel with the aim of producing a metallically clean surface. The protective passive layer can only be formed on such a metallically clean surface.
Scale and annealing colours are not passive layers and hence represent potential points for attack by corrosion.
The same applies to ferritic deposits, which are inherently almost unavoidable in production.
The pickling process removes all contaminants by chemical means (with acids) and lead to metallically clean surfaces.