Sep 21, 2016
Table of contents:
Over the last couple of weeks we’ve covered many of the fascinating aspects of Elixir, Erlang, and OTP that allow us to build highly available, fault tolerant applications.
First we looked at using Agents as a way to store state in server processes.
Next we looked at understanding GenServer and how we can use the tried and tested GenServer
module to build “generic server” processes.
A couple of weeks ago we looked at using Mix to organise our Elixir projects. Mix is a set of developer tools that ship with Elixir for organising, compiling, and testing your code.
Finally we looked at Supervisors. Supervisors are special processes that have the single responsibility of “supervising” other processes. Elixir applications are built in the form of supervision trees, and so supervisors play a critical role in allowing Elixir applications to recover from failure.
Whilst we have covered each of these components in isolation, I believe one of the best ways of learning is repeated practice. In today’s tutorial we are going to build a Casino using everything we have learned over the past couple of weeks.
So let’s shuffle up and deal!
So obviously we can’t build an actual working Casino in a single tutorial. Instead we’re going to focus on building out a supervision tree using the components we have been looking at over the last couple of weeks.
First up we need a way to register new players in the casino. Each player should have a name, a balance, and the ability to deposit money and make bets. We will need to supervise the players to keep the register up-to-date as players come and go.
Second we will build out the blackjack tables of the casino. We need a way of automatically generating the initial tables when the casino first opens, but we also need to add and remove tables as the demand fluctuates.
And of course, we need to build this Elixir application to deal with failure. If something goes wrong in one of the branches or leaves of the tree, we should be able to recover without disrupting the rest of the application.
So with our goals in mind, lets get on with building Philip’s Palace!
The first thing we need to do is to generate the application using Mix:
mix new casino -sup
When you run the command above in terminal, Mix will generate a new Elixir application called casino
. The --sup
flag is instructing Mix to create this application with a top level Supervisor ready to go.
By generating the application with a top level Supervisor we skip the requirement to create our own, and it means we will be able to start and stop the application as a single unit.
If you read Organising your Elixir project with Mix, you should be familiar with generating a new Elixir application. However, by using the --sup
flag there are a couple of things that are different.
First, the application/0
function in mix.exs
will already contain our application module:
def application do
[applications: [:logger], mod: {Casino, []}]
end
Second, the Casino
module is already using Application
and the top level Supervisor
is ready to go:
defmodule Casino do
use Application
def start(_type, _args) do
import Supervisor.Spec, warn: false
children = []
opts = [strategy: :one_for_one, name: Casino.Supervisor]
Supervisor.start_link(children, opts)
end
end
At this point you can fire up iex
to see the application running:
iex -S mix
For the remainder of this tutorial, whenever I mention using iex
you should start it as I have done in the command above to compile and load the application.
If you run the observer you will see that the Casino.Supervisor
is the root of the tree:
:observer.start()
As we progress through this tutorial I would recommend that you keep firing up iex
and running the observer to see how the tree is getting built. Being able to visualise exactly what the tree looks like really helped me to understand how to build Elixir applications.
The first aspect of the casino that we’re going to build will be to deal with the players. You can’t have a casino without people ready to lose money.
So first up we’re going to need a Player
process. Each process will represent one player in the casino, and the process should hold the current balance of the player.
Create a new directory under lib/casino
called players
. Next create a new file under the players
directory called player.ex
:
defmodule Casino.Players.Player do
end
We need a way to to start a process and hold the player’s current balance as it’s state. This is the perfect job for an Agent!
First up we need to implement the start_link/1
function:
def start_link(balance) do
Agent.start_link(fn -> balance end, [])
end
This function accepts the starting balance
of the player when the process is created. The balance is then used as the internal state of the process.
Next we can add a couple of functions to check the balance, deposit more money, or make a bet:
def balance(pid) do
Agent.get(pid, & &1)
end
def deposit(pid, amount) do
Agent.update(pid, &(&1 + amount))
end
def bet(pid, amount) do
Agent.update(pid, &(&1 - amount))
end
If you read Using Agents in Elixir this should hopefully be fairly straight forward for you.
At this point we can fire up iex
again to have a play with the new Player
process:
# Start the process
{:ok, pid} = Casino.Players.Player.start_link(100)
# Place some bets
Casino.Players.Player.bet(pid, 20)
Casino.Players.Player.bet(pid, 30)
# Deposit more money
Casino.Players.Player.deposit(pid, 100)
# Check the balance
Casino.Players.Player.balance(pid)
Now that we have the player process we need a way to create new player processes. If you are familiar with other programming languages you might recognise this as the factory pattern.
The factory in this application is going to be a supervisor. Create a new file called player_supervisor.ex
under the players
directory:
defmodule Casino.Players.PlayerSupervisor do
use Supervisor
def start_link do
Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
end
def init(:ok) do
children = [
worker(Casino.Players.Player, [], restart: :temporary)
]
supervise(children, strategy: :simple_one_for_one)
end
def new_player(balance) do
Supervisor.start_child(Casino.Players.PlayerSupervisor, [balance])
end
end
This supervisor is using the :simple_one_for_one
strategy to create child processes. This strategy is perfect for when the number of child processes should be dynamic, as is the case for creating new players as they enter the casino.
I’m also passing the restart
option as :temporary
for the Casino.Players.Player
worker. This means the supervisor will not restart the process if it dies. Usually the supervisor will be responsible for restarting processes automatically, but in this case I don’t mind if the player process dies because it just means that the player got kicked out of the casino.
And finally I’ve added new_player/1
as the “factory function” to create a new player. This function accepts the starting balance and then it creates a new player using the specification of the supervisor.
If you fire up iex
again you should be able to use the supervisor to create a new player:
# Start the supervisor
Casino.Players.PlayerSupervisor.start_link()
# Create a new player
Casino.Players.PlayerSupervisor.new_player(1000)
Note, because we named the PlayerSupervisor
, we don’t need to use the pid that was returned when the process started.
The final part of the player functionality of the casino is to create a player server that acts as the public interface. This server process is responsible for keeping track of the current active players, adding new players, and removing players when they leave:
defmodule Casino.Players.Server do
end
This is a “generic” server process because we need to hold state, but also deal with a couple of other responsibilities. Therefore this is a perfect opportunity to use GenServer
:
defmodule Casino.Players.Server do
use GenServer
end
First up we can implement the start_link
function:
def start_link do
GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
end
Once again I’m setting the name
of this process as there is only going to be one. This also makes it easier to work with because we don’t have to use the pid.
When the start_link
function is called on the client side, the init/1
function will automatically be called on the server side. This is our opportunity to set up the state of this process:
def init(:ok) do
players = %{}
refs = %{}
{:ok, {players, refs}}
end
The state of the process is a two element tuple where both elements are empty maps.
This server process is going to be responsible for keeping a record of the players in the casino. The player details will be stored in the players
map.
Each player is a process and sometimes processes die. When a player process dies we need to update our players list to remove that player. In order to do that we need to monitor the player processes and then listen for the process sending a message when it dies. We will store the references to each player process in the refs
map.
Now that we have the process and it’s state set up, we can add the functionality.
First up is the add/2
function that accepts the player’s name and their starting balance:
def add(name, balance) when is_binary(name) and is_number(balance) do
GenServer.cast(__MODULE__, {:add, name, balance})
end
In this function I’m guarding to make sure the name
is a binary and the balance
is a number. I’m then sending a cast
request to the server.
On the server side I will accept this request using handle_cast/2
function and pattern matching on the arguments of the request:
def handle_cast({:add, name, balance}, {players, refs}) do
end
The second argument of this function is the current state of the process.
The first thing I will do is to create a new player process using the PlayerSupervisor
from earlier:
{:ok, pid} = Casino.Players.PlayerSupervisor.new_player(balance)
With the player process created and the pid captured I can now monitor the process to get the reference:
ref = Process.monitor(pid)
This means if the player process dies, this server process will receive a message with the details of the process that died.
Next we need to generate an id for the player. The first player should have the id of 1
and each time we add a new player the id should auto increment:
id = auto_increment(players)
We can deal with this functionality by using a multi-clause function. When there are no players in the casino, the first player should automatically be set to 1
:
defp auto_increment(players) when players == %{}, do: 1
When there are existing players in the casino, we can get the next id in the sequence by converting the keys of the map to a list, getting the last key, and then adding 1:
defp auto_increment(players) do
Map.keys(players)
|> List.last()
|> Kernel.+(1)
end
Back in the handle_cast
function, we can save the reference and the player to their respective maps, and then return the correct :noreply
response from the function:
refs = Map.put(refs, ref, id)
players = Map.put(players, id, {name, pid, ref})
{:noreply, {players, refs}}
The second element of the :noreply
tuple is the new state of the process.
Here is that function in full:
def handle_cast({:add, name, balance}, {players, refs}) do
{:ok, pid} = Casino.Players.PlayerSupervisor.new_player(balance)
ref = Process.monitor(pid)
id = auto_increment(players)
refs = Map.put(refs, ref, id)
players = Map.put(players, id, {name, pid, ref})
{:noreply, {players, refs}}
end
Next up we can add a function to remove a player by their id:
def remove(id) do
GenServer.cast(__MODULE__, {:remove, id})
end
Again this function is making a cast
request to the server, passing the id of the player that should be removed.
On the server side we can handle this request using another handle_cast/2
function:
def handle_cast({:remove, id}, {players, refs}) do
{{_name, pid, _ref}, players} = Map.pop(players, id)
Process.exit(pid, :kill)
{:noreply, {players, refs}}
end
First we get the player from the players
map using the given id. Next we tell the player process to exit.
Finally we once again return a :noreply
response, passing the state as the second element of the tuple.
The final function of the public API of this server process is to list out all of the active players:
def list do
GenServer.call(__MODULE__, {:list})
end
This time we need a response from the server and so we need to make a call
request.
On the server side we accept this request using a handle_call/3
function:
def handle_call({:list}, _from, {players, _refs} = state) do
list =
Enum.map(players, fn {id, {name, pid, _ref}} ->
%{id: id, name: name, balance: Casino.Players.Player.balance(pid)}
end)
{:reply, list, state}
end
In this function I’m mapping over the map of players to convert it into a list of maps containing the player’s id, name, and current balance.
The last thing we need to add to this server process is the function to listen for processes that have died so we can remove them from the server state:
def handle_info({:DOWN, ref, :process, _pid, _reason}, {players, refs}) do
{id, refs} = Map.pop(refs, ref)
players = Map.delete(players, id)
{:noreply, {players, refs}}
end
def handle_info(_msg, state) do
{:noreply, state}
end
Here I’m using the handle_info
function to listen for the :DOWN
message that will be sent when one of the player processes we’re monitoring dies.
When we receive this message we can use the reference to update the players
and refs
maps and then return the new state.
Finally when you implement the handle_info
function, you also need to provide the default implementation.
Now that we have the player server process finished we can take it for a spin in iex
:
# Start the player supervisor
Casino.Players.PlayerSupervisor.start_link()
# Start the server
Casino.Players.Server.start_link()
# Add a player
Casino.Players.Server.add("Philip", 100)
# Add another player
Casino.Players.Server.add("Jane", 250)
# List the active players
Casino.Players.Server.list()
# Remove a player
Casino.Players.Server.remove(1)
# Check the list of active players
Casino.Players.Server.list()
Try opening the :observer
too so you can see that as player processes are created they are joined to the PlayerSupervisor
as children. If you kill a player process and then check the list of active players you will also see that the list is automatically updated.
As you can see, we have successfully built out the players functionality of the casino. However, one annoying thing is that we need to manually start the player supervisor and server in order to use them. Also if either of these processes die we need to be able to recover from the problem.
As you have probably guessed, this means we need to add another supervisor. Create a new file called players_supervisor.ex
under the casino
directory:
defmodule Casino.PlayersSupervisor do
use Supervisor
def start_link do
Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
end
def init(:ok) do
children = [
worker(Casino.Players.Server, []),
supervisor(Casino.Players.PlayerSupervisor, [])
]
supervise(children, strategy: :rest_for_one)
end
end
This supervisor is the root of the players functionality. As you can see from the children
list, this supervisor is responsible for starting the worker server and the supervisor so we don’t have to do that manually.
I’ve also chosen a strategy of :rest_for_one
. This means if the worker server dies, everything after it will also be restarted, but if the supervisor dies, the worker will not die. This makes sense because if the worker dies, we need to kill all of the player processes, but if the supervisor dies there is no need to kill the worker.
Finally I’m going to update the casino.ex
file to add the PlayersSupervisor
supervisor to the list of children:
defmodule Casino do
use Application
def start(_type, _args) do
import Supervisor.Spec, warn: false
children = [
supervisor(Casino.PlayersSupervisor, [])
]
opts = [strategy: :one_for_one, name: Casino.Supervisor]
Supervisor.start_link(children, opts)
end
end
Now when you fire up the application in iex
all of the player functionality should be automatically started and ready to go. What’s more, if there is a problem in the player functionality, the problem will be isolated to only that part of the casino and it won’t touch the rest of the application.
If you start :observer
and click on the Applications
tab you should see the tree of processes we have created so far. Try killing random processes to see how the application will automatically recover.
Give yourself a pat on the back, you have just created your first supervision tree in Elixir!
The next stage of building “Philip’s Palace” is to create the blackjack gaming section. In theory, blackjack would be just one of many games in the casino, and so we can represent this idea by creating a new games
directory under lib
and then another directory called blackjack
under the games
directory.
Each blackjack table should be represented as a process and so the first thing we need to do is to create a new file called table.ex
under the blackjack
directory:
defmodule Casino.Games.Blackjack.Table do
def start_link do
Agent.start_link(fn -> [] end, [])
end
end
The point of this tutorial is create the supervision tree, rather than actually create the casino. Therefore I’m just going to create the table process without any functionality. If you want to expand upon this tutorial you can add the more functionality to the table process so that players can start to play blackjack.
Next we need to create a supervisor that will act as the factory for creating new table processes:
defmodule Casino.Games.Blackjack.TableSupervisor do
use Supervisor
def start_link do
Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
end
def init(:ok) do
children = [
worker(Casino.Games.Blackjack.Table, [], restart: :temporary)
]
supervise(children, strategy: :simple_one_for_one)
end
def start_table do
Supervisor.start_child(Casino.Games.Blackjack.TableSupervisor, [])
end
end
As you can see this is pretty much the same as the player version. We once again use restart: :temporary
so each table won’t be restarted by the supervisor, and we use the :simple_one_for_one
strategy so we can dynamically control how many table processes are created. Finally we add the start_table/0
factory function for creating new tables.
Next up we need to create the public interface for the blackjack functionality of the casino. The server is going to be responsible for creating the initial blackjack tables when the casino first opens, as well as accepting requests to open more tables or close existing tables.
Once again because this is a “generic server” so we will use GenServer
:
defmodule Casino.Games.Blackjack.Server do
use GenServer
# Client
def start_link(state) do
GenServer.start_link(__MODULE__, state, name: __MODULE__)
end
# Server
def init(state) do
send(self, {:start})
{:ok, state}
end
end
Here I’m setting up a basic GenServer
and giving it a name. The state in this server is the number of active tables. When the server is started, it is given the state and it is the server’s responsibility to start that number of tables. As tables are opened and closed to meet demand, the server will increment and decrement this number.
An interesting aspect of this server is that we need to set up the initial tables. To ensure that the server process starts quickly, instead of dealing with that in the init/1
function, we can send a message to the process to do it asynchronously.
To accept this message we need to use the handle_info
function:
def handle_info({:start}, state) do
open_table(state)
{:noreply, state}
end
The <code>state</code> is going to be a number that is going to be greater than 0. To handle creating the tables we can pass the state to a function called <code>open_table/1</code>:
defp open_table(n) when is_number(n) and n <= 1 do
start_table
end
defp open_table(n) when is_number(n) do
start_table
open_table(n - 1)
end
defp start_table do
{:ok, pid} = Casino.Games.Blackjack.TableSupervisor.start_table()
Process.monitor(pid)
end
If the number is 1 we can simply call the start_table
function once. If the number is greater than 1 we can recursively call the open_table
function.
The start_table/0
function simple starts a new table using the TableSupervisor
and then monitors the pid of the new process.
At this point we can also handle the :DOWN
message that will be sent if one of the table processes dies:
def handle_info({:DOWN, _ref, :process, _pid, _reason}, state) do
{:noreply, state - 1}
end
def handle_info(_msg, state) do
{:noreply, state}
end
We aren’t keeping a list of tables in this server process so the only thing we need to do when a table process dies is to reduce the state
by 1.
Next up we can start to add the public API of this server process.
First we will add a function for adding 1 or more new tables to satisfy the demand:
def add_table(count \\ 1) do
GenServer.cast(__MODULE__, {:add, count})
end
This function accepts a number for the count of tables to create, but if a number is not supplied it will default to 1.
On the server side we can use the open_table/1
function to add a new table process and then return the state
plus the count
of new tables:
def handle_cast({:add, count}, state) do
open_table(count)
{:noreply, state + count}
end
Next we can add a function to remove a table:
def remove_table do
GenServer.cast(__MODULE__, {:remove})
end
On the server side we need to close a table and then update the state:
def handle_cast({:remove}, state) do
close_table(state)
{:noreply, state}
end
We can deal with closing tables using the close_table/1
multi-clause function.
Firstly, if there are currently no active tables we can just return 0:
defp close_table(state) when state == 0, do: 0
Otherwise we can get the current active children from the TableSupervisor
, find the last child and then remove it:
defp close_table(state) when is_number(state) do
Supervisor.which_children(Casino.Games.Blackjack.TableSupervisor)
|> List.last()
|> close_table
end
defp close_table({_, pid, _, _}) when is_pid(pid) do
Supervisor.terminate_child(Casino.Games.Blackjack.TableSupervisor, pid)
end
In these two functions I’m using the which_children
function of the Supervisor
module to get the current active children.
I’m then passing the tuple of the last child and using the pid to terminate the process.
Finally I’m returning the state.
Last up I’ll add a function to return the count of active tables:
def count_tables do
GenServer.call(__MODULE__, {:count})
end
To handle this request we simply need to return the current state:
def handle_call({:count}, _from, state) do
{:reply, state, state}
end
Now that we have the blackjack table, supervisor, and server processes in place, the final thing to do is to create the final supervisors for this section of the casino.
First up we need to create the blackjack supervisor, which will be responsible for starting the table supervisor and the server:
defmodule Casino.Games.Blackjack.Supervisor do
use Supervisor
def start_link do
Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
end
def init(:ok) do
children = [
supervisor(Casino.Games.Blackjack.TableSupervisor, []),
worker(Casino.Games.Blackjack.Server, [4])
]
supervise(children, strategy: :one_for_all)
end
end
When the blackjack server is started, I’m passing 4
as the initial state and so 4 tables should be automatically created when the casino opens for business. Alternatively you could set this as a configuration option or an environment variable.
Next up we need to create a games supervisor, which will be responsible for starting each game section of the casino. We only have a blackjack section at the minute, but as you can imagine, the casino is likely to add more games:
defmodule Casino.GamesSupervisor do
use Supervisor
def start_link do
Supervisor.start_link(__MODULE__, :ok, name: __MODULE__)
end
def init(:ok) do
children = [
supervisor(Casino.Games.Blackjack.Supervisor, [])
]
supervise(children, strategy: :one_for_one)
end
end
Here I’m using the :one_for_one
strategy so each gaming section is not affected by crashes in the other gaming sections.
And finally we can add the games supervisor to the top level supervisor so it is automatically started when the application is started:
defmodule Casino do
use Application
def start(_type, _args) do
import Supervisor.Spec, warn: false
children = [
supervisor(Casino.PlayersSupervisor, []),
supervisor(Casino.GamesSupervisor, [])
]
opts = [strategy: :one_for_one, name: Casino.Supervisor]
Supervisor.start_link(children, opts)
end
end
Again I’m using the :one_for_one
strategy so each section is independent from the rest.
As one final touch, I’m going to add a couple of helper functions to the top level Casino
module to make interacting with the casino easier:
def add_player(name, balance) do
Casino.Players.Server.add(name, balance)
end
def remove_player(id) do
Casino.Players.Server.remove(id)
end
def list_players do
Casino.Players.Server.list()
end
def add_blackjack_table(count \\ 1) do
Casino.Games.Blackjack.Server.add_table(count)
end
def remove_blackjack_table do
Casino.Games.Blackjack.Server.remove_table()
end
def count_blackjack_tables do
Casino.Games.Blackjack.Server.count_tables()
end
Now if you fire up iex
and the observer, we can take the casino for a spin:
# Add a new player
Casino.add_player("Philip", 100)
# Add another player
Casino.add_player("Jane", 250)
# List all of the players
Casino.list_players()
# Remove a player
Casino.remove_player(2)
# Count the active blackjack tables
Casino.count_blackjack_tables()
# Add 3 more blackjack tables
Casino.add_blackjack_table(3)
# Count the blackjack tables again
Casino.count_blackjack_tables()
# Remove a blackjack table
Casino.remove_blackjack_table()
# Count the blackjack tables one last time
Casino.count_blackjack_tables()
Phew, that was a long tutorial! Well done for sticking it out and getting to the end!
Building applications in Elixir and Erlang is probably very different if you are coming from just about any other programming language. The whole notion of designing an application around supervision trees was a whole new concept to me when I started to explore Elixir.
One of the things that really helped my get my head around this idea was to build out toy applications like we have done so today.
I really do thing the best way to learn a concept is to try it out in the real world. There is nothing quite like writing code to understand something that seems so abstract when written down as words.
I hope today’s tutorial has clarified a few of the key concepts of building applications in Elixir, and I hope it has inspired you to start building your own toy supervision applications. You can find the code from today’s tutorial on GitHub.