Download the full-sized PDF of Optimal Mechanisms for Machine Learning: A Game-Theoretic Approach to Designing Machine Learning CompetitionsDownload the full-sized PDF



Permanent link (DOI):


Export to: EndNote  |  Zotero  |  Mendeley


This file is in the following communities:

Graduate Studies and Research, Faculty of


This file is in the following collections:

Theses and Dissertations

Optimal Mechanisms for Machine Learning: A Game-Theoretic Approach to Designing Machine Learning Competitions Open Access


Other title
machine learning
game theory
mechanism design
Type of item
Degree grantor
University of Alberta
Author or creator
Ajallooeian, Mohammad Mahdi
Supervisor and department
Szepesvári, Csaba (Computing Science)
Examining committee member and department
György, András (Computing Science)
Jinag, Hai (Electrical and Computer Engineering)
Schuurmans, Dale (Computing Science)
Bulitko, Vadim (Computing Science)
Department of Computing Science

Date accepted
Graduation date
Master of Science
Degree level
In this thesis we consider problems where a self-interested entity, called the principal, has private access to some data that she wishes to use to solve a prediction problem by outsourcing the development of the predictor to some other parties. Assuming the principal, who needs the machine learning solution, and the potential providers of the solution are two independent, self-interested agents, which is the case for many real-world situations, this then becomes a game-theoretic problem. We propose mathematical models for variants of this problem by borrowing techniques from the literature of mechanism design and provide principled solutions. We consider experimental design when there are multiple self-interested agents involved in developing a solution for a machine learning problem. A first case is when there is a public competition, each agent offers a single solution and solutions are available off-the-shelf to the agents: there is no development cost included. The problem considered is to find a set of payment rules that guarantees to maximize the profit of the principal on expectation even when the developers are self-interested. The solution depends on the distribution of the skill-level of developers available, which is assumed to be known. To deal with our problem, the standard mechanism design techniques are revisited and extended in a number of ways. In particular, a general approach is given that allows the design of payment rules (more generally, mechanisms) when such payment rules must depend on some quantity that becomes known only after the mechanism is executed. This extension plays a key role in our solution to the machine learning payment-rule design problem, where data must be kept private (otherwise the developers could submit “overfitting” predictors), yet the principal’s profit (and thus the payment) should depend on the performance of the predictor chosen on theprivate data. Then, we address other interesting variants of the problem and provide solutions : when a single developer can submit multiple solutions, and when the solution is to be developed in multiple stages, or when the development cost is non-zero.
Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.
Citation for previous publication

File Details

Date Uploaded
Date Modified
Audit Status
Audits have not yet been run on this file.
File format: pdf (Portable Document Format)
Mime type: application/pdf
File size: 662401
Last modified: 2015:10:12 16:47:19-06:00
Filename: Ajallooeian_Mohammad Mahdi_Spring 2013.pdf
Original checksum: 3e7cd68dd6f1356c3728dfb5e5fe62b2
Well formed: false
Valid: false
Status message: No document catalog dictionary offset=0
Activity of users you follow
User Activity Date