You might think that any cloth found in the desert could be called desert cloth, but you’d just be arguing semantics and that won’t win you many friends at parties. No, the desert cloth we refer to here is the sort of thing woven from desert weed and cacti fibres. It is forged of the desert, not simply found within it.
That said, explaining this at parties probably still won’t make you many friends.