Table of Contents¶
An example of a small Multi-Player simulation, with rhoRand and Selfish, for different algorithms¶
First, be sure to be in the main folder, or to have SMPyBandits installed, and import EvaluatorMultiPlayers from Environment package:
!pip install SMPyBandits watermark
%load_ext watermark
%watermark -v -m -p SMPyBandits -a "Lilian Besson"
# Local imports
from SMPyBandits.Environment import EvaluatorMultiPlayers, tqdm
We also need arms, for instance Bernoulli-distributed arm:
# Import arms
from SMPyBandits.Arms import Bernoulli
And finally we need some single-player and multi-player Reinforcement Learning algorithms:
# Import algorithms
from SMPyBandits.Policies import *
from SMPyBandits.PoliciesMultiPlayers import *
# Just improving the ?? in Jupyter. Thanks to https://nbviewer.jupyter.org/gist/minrk/7715212
from __future__ import print_function
from IPython.core import page
def myprint(s):
try:
print(s['text/plain'])
except (KeyError, TypeError):
print(s)
page.page = myprint
For instance, this imported the Thompson algorithm:
Thompson?
As well as the rhoRand and Selfish multi-player policy:
rhoRand?
Selfish?
We also need a collision model. The usual ones are defined in the CollisionModels package, and the only one we need is the classical one, where two or more colliding users don't receive any rewards.
# Collision Models
from SMPyBandits.Environment.CollisionModels import onlyUniqUserGetsReward
onlyUniqUserGetsReward?
Creating the problem¶
Parameters for the simulation¶
- $T = 10000$ is the time horizon,
- $N = 100$ is the number of repetitions (should be larger to have consistent results),
- $M = 2$ is the number of players,
N_JOBS = 4is the number of cores used to parallelize the code.
HORIZON = 10000
REPETITIONS = 100
NB_PLAYERS = 2
N_JOBS = 1
collisionModel = onlyUniqUserGetsReward
Three MAB problems with Bernoulli arms¶
We consider in this example $3$ problems, with Bernoulli arms, of different means.
- The first problem is very easy, with two good arms and three arms, with a fixed gap $\Delta = \max_{\mu_i \neq \mu_j}(\mu_{i} - \mu_{j}) = 0.1$.
- The second problem is as easier, with a larger gap.
- Third problem is harder, with a smaller gap, and a very large difference between the two optimal arms and the suboptimal arms.
Note: right now, the multi-environments evaluator does not work well for MP policies, if there is a number different of arms in the scenarios. So I use the same number of arms in all the problems.
ENVIRONMENTS = [ # 1) Bernoulli arms
{ # Scenario 1 from [Komiyama, Honda, Nakagawa, 2016, arXiv 1506.00779]
"arm_type": Bernoulli,
"params": [0.3, 0.4, 0.5, 0.6, 0.7]
},
{ # Classical scenario
"arm_type": Bernoulli,
"params": [0.1, 0.3, 0.5, 0.7, 0.9]
},
{ # Harder scenario
"arm_type": Bernoulli,
"params": [0.005, 0.01, 0.015, 0.84, 0.85]
}
]
Some RL algorithms¶
We will use Thompson Sampling, $\mathrm{UCB}_1$ and $\mathrm{kl}$-$\mathrm{UCB}$, using two different decentralized policy:
rhoRand: each player starts with a rank $1$, and on collision it selects a new rank $r \sim U([1,M])$. Instead of aiming at the best arm, each player aims at the $r$-th best arm (as given by its UCB index, or posterior sampling if Thompson sampling).Selfish: each player runs a classical RL algorithm, on the joint reward $\tilde{r}$: $$ \tilde{r}_j(t) = r_j(t) \times (1 - \eta_j(t)) $$ where $r_j(t)$ is the reward from the sensing of the $j$-th arm at time $t \geq 1$ and $\eta_j(t)$ is a boolean indicator saying that there were a collision on that arm at time $t$. In other words, if a player chose arm $j$ and was alone on it, it receives $r_j(t)$, and if it was not alone, all players who chose arm $j$ receive $0$.
nbArms = len(ENVIRONMENTS[0]['params'])
nbArms
SUCCESSIVE_PLAYERS = [
# UCB alpha=1
rhoRand(NB_PLAYERS, nbArms, UCBalpha, alpha=1).children,
Selfish(NB_PLAYERS, nbArms, UCBalpha, alpha=1).children,
# Thompson Sampling
rhoRand(NB_PLAYERS, nbArms, Thompson).children,
Selfish(NB_PLAYERS, nbArms, Thompson).children,
# Thompson Sampling
rhoRand(NB_PLAYERS, nbArms, klUCB).children,
Selfish(NB_PLAYERS, nbArms, klUCB).children,
]
SUCCESSIVE_PLAYERS
The mother class in this case does not do any centralized learning, as Selfish and rhoRand are fully decentralized scenario.
OneMother = SUCCESSIVE_PLAYERS[0][0].mother
OneMother
OneMother.nbArms
OnePlayer = SUCCESSIVE_PLAYERS[0][0]
OnePlayer.nbArms
Complete configuration for the problem:
configuration = {
# --- Duration of the experiment
"horizon": HORIZON,
# --- Number of repetition of the experiment (to have an average)
"repetitions": REPETITIONS,
# --- Parameters for the use of joblib.Parallel
"n_jobs": N_JOBS, # = nb of CPU cores
"verbosity": 6, # Max joblib verbosity
# --- Collision model
"collisionModel": onlyUniqUserGetsReward,
# --- Arms
"environment": ENVIRONMENTS,
# --- Algorithms
"successive_players": SUCCESSIVE_PLAYERS,
}
Creating the EvaluatorMultiPlayers objects¶
We will need to create several objects, as the simulation first runs one policy against each environment, and then aggregate them to compare them.
%%time
N_players = len(configuration["successive_players"])
# List to keep all the EvaluatorMultiPlayers objects
evs = [None] * N_players
evaluators = [[None] * N_players] * len(configuration["environment"])
for playersId, players in tqdm(enumerate(configuration["successive_players"]), desc="Creating"):
print("\n\nConsidering the list of players :\n", players)
conf = configuration.copy()
conf['players'] = players
evs[playersId] = EvaluatorMultiPlayers(conf)
Solving the problem¶
Now we can simulate the $2$ environments, for the successive policies. That part can take some time.
%%time
for playersId, evaluation in tqdm(enumerate(evs), desc="Policies"):
for envId, env in tqdm(enumerate(evaluation.envs), desc="Problems"):
# Evaluate just that env
evaluation.startOneEnv(envId, env)
# Storing it after simulation is done
evaluators[envId][playersId] = evaluation
Plotting the results¶
And finally, visualize them, with the plotting method of a EvaluatorMultiPlayers object:
def plotAll(evaluation, envId):
evaluation.printFinalRanking(envId)
# Rewards
evaluation.plotRewards(envId)
# Fairness
evaluation.plotFairness(envId, fairness="STD")
# Centralized regret
evaluation.plotRegretCentralized(envId, subTerms=True)
#evaluation.plotRegretCentralized(envId, semilogx=True, subTerms=True)
# Number of switches
#evaluation.plotNbSwitchs(envId, cumulated=False)
evaluation.plotNbSwitchs(envId, cumulated=True)
# Frequency of selection of the best arms
evaluation.plotBestArmPulls(envId)
# Number of collisions
evaluation.plotNbCollisions(envId, cumulated=False)
evaluation.plotNbCollisions(envId, cumulated=True)
# Frequency of collision in each arm
evaluation.plotFrequencyCollisions(envId, piechart=True)
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (12.4, 7)
First problem¶
$\mu = [0.3, 0.4, 0.5, 0.6, 0.7]$ was an easy Bernoulli problem.
for playersId in tqdm(range(len(evs)), desc="Policies"):
evaluation = evaluators[0][playersId]
plotAll(evaluation, 0)
Second problem¶
$\mu = [0.1, 0.3, 0.5, 0.7, 0.9]$ was an easier Bernoulli problem, with larger gap $\Delta = 0.2$.
for playersId in tqdm(range(len(evs)), desc="Policies"):
evaluation = evaluators[1][playersId]
plotAll(evaluation, 1)
Third problem¶
$\mu = [0.005, 0.01, 0.015, 0.84, 0.85]$ is an harder Bernoulli problem, as there is a huge gap between suboptimal and optimal arms.
for playersId in tqdm(range(len(evs)), desc="Policies"):
evaluation = evaluators[2][playersId]
plotAll(evaluation, 2)
Comparing their performances¶
def plotCombined(e0, eothers, envId):
# Centralized regret
e0.plotRegretCentralized(envId, evaluators=eothers)
# Fairness
e0.plotFairness(envId, fairness="STD", evaluators=eothers)
# Number of switches
e0.plotNbSwitchsCentralized(envId, cumulated=True, evaluators=eothers)
# Number of collisions - not for Centralized* policies
e0.plotNbCollisions(envId, cumulated=True, evaluators=eothers)
N = len(configuration["environment"])
for envId, env in enumerate(configuration["environment"]):
e0, eothers = evaluators[envId][0], evaluators[envId][1:]
plotCombined(e0, eothers, envId)
That's it for this demo!