MASS: Masked Sequence to Sequence Pre-training for Language Generation
MASS is a novel pre-training method for sequence to sequence based language generation tasks. It randomly masks a sentence fragment in the encoder, and then predicts it in the decoder.