Zillow Group Inc. is calling it quits on the home-flipping business, while disclosing losses of more than $550 million on sales this year for which the company paid too much.