Losing Weight Know Your Labels