While these seq2seq models were initially

sultanamimi8889

Developed using recurrent neural networks, Transformer encoder-decoder models have recently come into vogue as they are more effective at modeling the dependencies present in the long sequences encountered in summarization. Transformer models combined with self-supervised pre-training (e.g., BERT, GPT-2, RoBERTa, XLNet, ALBERT, T5, ELECTRA) have proven to be a powerful framework for producing general language learning , achieving very high performance when queried on a wide range of linguistic tasks.

Until recently, so-called “self-monitored” objectives – used Bosnia and Herzegovina Mobile Number List in the pre-training phase – have been somewhat agnostic; Recently, however, scholars have wondered whether better performance could be obtained if the self-monitored goal more closely reflected the final task. With PEGASUS (which will appear at the 2020 International Conference on Machine Learning) a self-supervised pre-training objective (called gap sentence generation) has been designed for Transformer encoder-decoder models useful for improving commissioning performance to point on abstract summaries.



A self-monitored objective for the summary The PEGASUS developers' hypothesis is that the closer the pre-training self-surveillance goal is to the final downstream task, the better the tuning performance will be. In the pre-training phase of PEGASUS several entire sentences are removed from documents and the model is tasked with recovering them. An example of pre-training input is a document with missing sentences, while the output consists of the missing sentences concatenated together.