The building society failed to monitor its customers' activity effectively between 2016 and 2021, the regulator says.