Corresponding optimal action with respect to that MDP. In practice, the number of sampled models is determined dynamically with a parameter . The re-sampling frequency depends on a parameter . Tested values: ? 2 1.0, 1e – 1, 1e – 2, 1e – 3, 1e – 4, 1e – 5, 1e – 6, ? 2 9, 7, 5, 3, 1, 1e – 1, 1e – 2, 1e – 3n1e – 4, 1e – 5, 1e – 6. 5.1.8 BEB. The Bayesian Exploration Bonus (BEB) [14] is a Bayesian RL algorithm which builds, at each time-step t, the expected MDP given the current posterior. Before solving this MDP, it computes a new reward function r ? ; u; y??rM ; u; y?? ?b , where c ?BEB cdenotes the number of times transition < x, u, y > has been observed at time-step t. This algorithm solves the mean MDP of the current posterior, in which we replaced M(? ? ? by r ? ; ? , and applies its optimal policy on the current MDP for one step. The bonus is a BEBPLOS ONE | DOI:10.1371/journal.pone.0157088 June 15,11 /Benchmarking for Bayesian Reinforcement LearningTable 1. Influence of the algorithm and their parameters on the offline and online phases duration. Offline phase duration Random -Greedy OPPS-DS BAMCP BFS3 SBOSS Almost GDC-0084 chemical information instantaneous. Almost instantaneous. Varies proportionally to . Almost instantaneous. Almost instantaneous. Almost instantaneous. Online phase duration Almost instantaneous. Varies in inverse proportion to . Can vary a lot from one step to another. Varies proportionally to the number of features implied in the selected E/E Vesatolimod mechanism of action strategy. Varies proportionally to K and depth. Varies proportionally to K, C and depth. Varies in inverse proportion to and . Can vary a lot from one step to another, with a general decreasing tendency. Constant.BEBAlmost instantaneous.doi:10.1371/journal.pone.0157088.tparameter controlling the E/E balance. BEB comes with theoretical guarantees of convergence towards Bayesian optimality. Tested values: ? 2 0.25, 0.5, 1, 1.5, 2, 2.5, 3, 4, 8, 16. 5.1.9 Computation times variance. Each algorithm has one or more parameters that can affect the number of sampled transitions from a given state, or the length of each simulation. This, in turn, impacts the computation time requirement at each step. Hence, for some algorithms, no choice of parameters can bring the computation time below or over certain values. In other words, each algorithm has its own range of computation time. Note that, for some methods, the computation time is influenced concurrently by several parameters. We present a qualitative description of how computation time varies as a function of parameters in Table 1.5.2 BenchmarksIn our setting, the transition matrix is the only element which differs between two MDPs drawn from the same distribution. For each < state, action > pair < x, u >, we define a Dirichlet distribution, which represents the uncertainty about the transitions occurring from < x, u >. A Dirich X ?let distribution is parameterised by a set of concentration parameters a??> 0; . . . ; a > 0. We gathered all concentration parameters in a single vector . Consequently, our MDP distributions are parameterised by M (the reward function) and several Dirichlet distributions, parameterised by . Such a distribution is denoted by prM ; ? In the Bayesian Reinforcement Learning community, these distributions are referred to as Flat Dirichlet Multinomial distributions (FDMs). We chose to study two different cases: ?Accurate case: the test distribution is fully known (p0 ???pM ??, M ?Inaccurate case: the.Corresponding optimal action with respect to that MDP. In practice, the number of sampled models is determined dynamically with a parameter . The re-sampling frequency depends on a parameter . Tested values: ? 2 1.0, 1e – 1, 1e – 2, 1e – 3, 1e – 4, 1e – 5, 1e – 6, ? 2 9, 7, 5, 3, 1, 1e – 1, 1e – 2, 1e – 3n1e – 4, 1e – 5, 1e – 6. 5.1.8 BEB. The Bayesian Exploration Bonus (BEB) [14] is a Bayesian RL algorithm which builds, at each time-step t, the expected MDP given the current posterior. Before solving this MDP, it computes a new reward function r ? ; u; y??rM ; u; y?? ?b , where c ?BEB cdenotes the number of times transition < x, u, y > has been observed at time-step t. This algorithm solves the mean MDP of the current posterior, in which we replaced M(? ? ? by r ? ; ? , and applies its optimal policy on the current MDP for one step. The bonus is a BEBPLOS ONE | DOI:10.1371/journal.pone.0157088 June 15,11 /Benchmarking for Bayesian Reinforcement LearningTable 1. Influence of the algorithm and their parameters on the offline and online phases duration. Offline phase duration Random -Greedy OPPS-DS BAMCP BFS3 SBOSS Almost instantaneous. Almost instantaneous. Varies proportionally to . Almost instantaneous. Almost instantaneous. Almost instantaneous. Online phase duration Almost instantaneous. Varies in inverse proportion to . Can vary a lot from one step to another. Varies proportionally to the number of features implied in the selected E/E strategy. Varies proportionally to K and depth. Varies proportionally to K, C and depth. Varies in inverse proportion to and . Can vary a lot from one step to another, with a general decreasing tendency. Constant.BEBAlmost instantaneous.doi:10.1371/journal.pone.0157088.tparameter controlling the E/E balance. BEB comes with theoretical guarantees of convergence towards Bayesian optimality. Tested values: ? 2 0.25, 0.5, 1, 1.5, 2, 2.5, 3, 4, 8, 16. 5.1.9 Computation times variance. Each algorithm has one or more parameters that can affect the number of sampled transitions from a given state, or the length of each simulation. This, in turn, impacts the computation time requirement at each step. Hence, for some algorithms, no choice of parameters can bring the computation time below or over certain values. In other words, each algorithm has its own range of computation time. Note that, for some methods, the computation time is influenced concurrently by several parameters. We present a qualitative description of how computation time varies as a function of parameters in Table 1.5.2 BenchmarksIn our setting, the transition matrix is the only element which differs between two MDPs drawn from the same distribution. For each < state, action > pair < x, u >, we define a Dirichlet distribution, which represents the uncertainty about the transitions occurring from < x, u >. A Dirich X ?let distribution is parameterised by a set of concentration parameters a??> 0; . . . ; a > 0. We gathered all concentration parameters in a single vector . Consequently, our MDP distributions are parameterised by M (the reward function) and several Dirichlet distributions, parameterised by . Such a distribution is denoted by prM ; ? In the Bayesian Reinforcement Learning community, these distributions are referred to as Flat Dirichlet Multinomial distributions (FDMs). We chose to study two different cases: ?Accurate case: the test distribution is fully known (p0 ???pM ??, M ?Inaccurate case: the.