Matrix methods for stochastic dynamic programming in ecology and evolutionary biology

• Author(s) / Creator(s)

  • Jody R. Reimer
  • Marc Mangel
  • Andrew E. Derocher
  • Mark A. Lewis

• 1. Organisms are constantly making tradeoffs. These tradeoffs may be behavioural (e.g., whether to focus on foraging or predator avoidance) or physiological (e.g., whether to allocate energy to reproduction or growth). Similarly, wildlife and fisheries managers must make tradeoffs while striving for conservation or economic goals (e.g., costs versus rewards). Stochastic dynamic programming (SDP) provides a powerful and flexible framework within which to explore these tradeoffs. A rich body of mathematical results on SDP exist but have received little attention in ecology and evolution.
  2. Using directed graphs—an intuitive visual model representation—we reformulated SDP models into matrix form. We synthesized relevant existing theoretical results which we then applied to two canonical SDP models in ecology and evolution. We applied these matrix methods to a simple illustrative patch choice example and an existing SDP model of parasitoid wasp behaviour.
  3. The proposed analytical matrix methods provide the same results as standard numerical methods as well as additional insights into the nature and quantity of other, nearly optimal, strategies, which we may also expect to observe in nature. The mathematical results highlighted in this work also explain qualitative aspects of model convergence. An added benefit of the proposed matrix notation is the resulting ease of implementation of Markov chain analysis (an exact solution for the realized states of an individual) rather than Monte Carlo simulations (the standard, approximate method). It also provides an independent validation method for other numerical methods, even in applications focused on short-term, non-stationary dynamics.
  4. These methods are useful for obtaining, interpreting, and further analysing model convergence to the optimal time-independent (i.e., stationary) decisions predicted by an SDP model. SDP is a powerful tool both for theoretical and applied ecology, and an understanding of the mathematical structure underlying SDP models can increase our ability to apply and interpret these models.

• Date created

  2019-01-01

• Subjects / Keywords

  • stationary decisions
  • Markov chain
  • optimality models
  • backwards induction
  • Markov decision process
  • value iteration
  • stochastic dynamic programming

• Type of Item

  Article (Draft / Submitted)

• DOI

  https://doi.org/10.7939/r3-jh8v-be98

• License

  Attribution-NonCommercial 4.0 International