Finding Surprisingly Frequent Patterns of Variable Lengths in Sequence Data

  • Author / Creator
    Sadoddin, Reza
  • We address the problem of finding ‘surprising’ patterns of variable length in sequence data, where a surprising pattern is defined as a subsequence of a longer sequence, whose observed frequency is statistically significant with respect to a given distribution.Finding statistically significant patterns in sequence data is the core task in some interesting applications such as Biologicial motif discovery and anomaly detection. We investigate the problem of ‘redundant patterns’, where the presence of few ‘true’ anomalous patterns in the data could cause a large number of highly-correlated patterns to stand statistically significant just because of those few anomalous patterns. Identifying ‘true’ anomalies in a set with many ‘redundant patterns’ can be challenging. Our approach to solving this problem is based on capturing the dependencies between patterns through an ‘explain’ relationship where a set of patterns can explain the statistical significance of another pattern. The ‘explain’ relationship allows us to address the problem of redundancy by choosing a few ‘core’ patterns which explain the significance of all other significant patterns. We propose a greedy algorithm for efficiently finding an approximate core pattern set of minimum size. To extend the utility of our method to a broader class of applications, the proposed framework is generalized by allowing the ‘surprising patterns’ to represent a class of subsequences with a certain amount of variation w.r.t a core pattern. Using both synthetic and real-world sequential data, chosen from different domains including Medicine, Computer Security, and Bioinformatics, we show that the proposed notion of core patterns very closely matches the notion of ‘true’ surprising patterns in data. We also compare our method with five other well-known anomaly detection techniques. The results show a better matching of our predictions with the ground truth compared to those of our comparison partners. When compared with 14 wellknown methods on the interesting application of the Biological motif discovery, and on a widely-used benchmark, our proposed method achieves better or comparable results in finding motifs, a special case of our surprising patterns.

  • Subjects / Keywords
  • Graduation date
  • Type of Item
  • Degree
    Doctor of Philosophy
  • DOI
  • License
    This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.
  • Language
  • Institution
    University of Alberta
  • Degree level
  • Department
    • Department of Computing Science
  • Supervisor / co-supervisor and their department(s)
    • Rafiei, Davood(Computing Science)
    • Sander, Joerg (Computing Science)
  • Examining committee members and their departments
    • Kurgan, Lukasz(Electrical & Computer Engineering)
    • Pei, Jian(Computer Science, Simon Fraser University)
    • Lin, Guohui (Computing Science )