Few-shot, Interpolation-based Style-conditioned Text Generation using LLMs

• Author / Creator

  Gaafar, Moemen K

• This thesis addresses the task of few-shot style-conditioned text generation using large language models (LLMs). We propose a novel, model-agnostic approach for adapting LLMs to arbitrary styles using a few text samples from a certain author. Instead of using pre-defined features, our method represents style directly in terms of model weights and employs a Variational Autoencoder (VAE) to construct a latent space of these weights, allowing for a generic style representation.

  We investigate whether style features can be generically extracted from LLM weights, if a salient latent space can effectively encode authorial styles, and whether an interpolation strategy can extract novel finetuned models for low-resource authors. Our approach is evaluated on three datasets, comparing it to naive finetuning and prompting techniques.

  Results show that our method outperforms and is more reliable than naive finetuning in low-resource settings based on automatic metrics. While our method outperforms prompting for some LLMs using a low number of text samples, its performance still does not consistently exceed that of prompting, especially as the number of available text samples increases. This work contributes to controllable text generation by introducing a weight space interpolation technique for few-shot style adaptation and demonstrating that model weights can directly represent text style, providing insights for future research in this area.

• Subjects / Keywords

  • Text Generation
  • Large Language Models
  • Style-conditioned
  • Few-shot
  • Interpolation-based

• Graduation date

  Fall 2024

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-j850-8h95

• License

  This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.