Non-Autoregressive Unsupervised Summarization with Length-Control Algorithms

• Author / Creator

  Liu, Puyuan

• Text summarization aims to generate a short summary for an input text and has extensive real-world applications such as headline generation.
  State-of-the-art summarization models are mainly supervised; they require large labeled training corpora and thus cannot be applied to less popular areas, e.g., less spoken languages, where paired data are rare.

  In this thesis, I present a non-autoregressive unsupervised summarization model, which does not require parallel data for training.
  Our approach first performs edit-based search towards a heuristically defined score and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search results.
  Further, we design two length-control algorithms for the model, which perform dynamic programming on the model output and are able to explicitly control the number of words and characters in the generated summary, respectively.
  Such length control is important for the summarization task, because the main evaluation metric for summarization systems, i.e., ROUGE score, is sensitive to the summary length, and because real-word applications generally involve length constraints.

  Experiments on two benchmark datasets show that our approach achieves state-of-the-art performance for unsupervised summarization, yet largely improves inference efficiency.
  Further, our length-control algorithms are able to perform
  length-transfer generation, i.e., generating summaries of different lengths than the training target.

• Subjects / Keywords

  • summarization
  • non-autoregressive
  • length-control

• Graduation date

  Fall 2022

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-g4s3-mf88

• License

  This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.