Contrastive Decoding for Concepts in the Brain

• Author / Creator

  Efird, Cory

• This thesis presents a novel data-driven approach for identifying categoryselective regions in the human brain that are consistent across multiple participants. By leveraging a massive fMRI dataset and a multi-modal (language
  and image) neural network (CLIP), we trained a highly accurate contrastive
  brain decoder to predict neural responses to naturalistic images in the human
  visual cortex. We then applied a novel adaptation of the DBSCAN clustering
  algorithm to identify clusters of voxels across multiple brains that decode similar concepts, which we term shared decodable concepts (SDCs). The SDCs
  are interpreted by identifying the closest embeddings to each cluster centroid
  and analyzing the associated images and text. In contrast to other methods,
  ours does not require registration to a template space, allowing us to maintain
  the unique functional layout of each participants brain. It also uncovers both
  activating and deactivating stimuli, highlighting the importance of both in
  understanding brain function. Our approach allowed us to uncover categoryselective areas for food, subcategories of bodies and places, color, numerosity,
  object size, softness, lighting conditions, and more, demonstrating the versatility and potential of our approach for exploring brain functions.

• Subjects / Keywords

  • Machine Learning
  • Neuroscience
  • Contrastive Learning

• Graduation date

  Fall 2024

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-hy39-3v32

• License

  This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.