ERA

Download the full-sized PDF of The Baseline Approach to Agent EvaluationDownload the full-sized PDF

Analytics

Share

Permanent link (DOI): https://doi.org/10.7939/R3XW4832B

Download

Export to: EndNote  |  Zotero  |  Mendeley

Communities

This file is in the following communities:

Graduate Studies and Research, Faculty of

Collections

This file is in the following collections:

Theses and Dissertations

The Baseline Approach to Agent Evaluation Open Access

Descriptions

Other title
Subject/Keyword
variance
agent
baseline
estimator
Type of item
Thesis
Degree grantor
University of Alberta
Author or creator
Davidson, Joshua
Supervisor and department
Bowling, Michael (Computing Science)
Examining committee member and department
Holte, Robert (Computing Science)
Hoover, Jim (Computing Science)
Department
Department of Computing Science
Specialization

Date accepted
2014-01-30T14:29:21Z
Graduation date
2014-06
Degree
Master of Science
Degree level
Master's
Abstract
Efficient, unbiased estimation of agent performance is essential for drawing statistically significant conclusions in multi-agent domains with high outcome variance. Naive Monte Carlo estimation is often insufficient, as it can require a prohibitive number of samples, especially when evaluating slow-acting agents. Classical variance reduction techniques typically require careful encoding of domain knowledge or are intrinsically complex. In this work, we introduce the baseline method of creating unbiased estimators for zero-sum, multi-agent high-variance domains. We provide two examples of estimators created using this approach, one that leverages computer agents in self-play, and another that utilizes existing player data. We show empirically that these baseline estimators are competitive with state-of-the-art techniques for efficient evaluation in variants of computer poker, a zero-sum domain with notably high outcome variance. Additionally, we demonstrate how simple, yet effective, baseline estimators can be created and deployed in domains where efficient evaluation techniques are currently non-existent.
Language
English
DOI
doi:10.7939/R3XW4832B
Rights
Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.
Citation for previous publication
Joshua Davidson, Christopher Archibald, and Michael Bowling. Baseline: Practical Control Variates for Agent Evaluation in Zero-Sum Domains. In AAMAS ’13 Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, pages 1005–1012, 2013.

File Details

Date Uploaded
Date Modified
2014-06-15T07:07:15.159+00:00
Audit Status
Audits have not yet been run on this file.
Characterization
File format: pdf (Portable Document Format)
Mime type: application/pdf
File size: 670909
Last modified: 2015:10:18 01:36:36-06:00
Filename: Davidson_Joshua_Spring2014.pdf
Original checksum: 424bc63241ddf34242ec51d04b057411
Well formed: false
Valid: false
Status message: No document catalog dictionary offset=0
Activity of users you follow
User Activity Date