cult3

What is ETS in Elixir?

Oct 05, 2016

Table of contents:

  1. What is ETS?
  2. Creating tables
  3. Adding data
  4. Reading data
  5. Deleting data and destroying tables
  6. What is DETS?
  7. Using ETS with GenServer
  8. Conclusion

If you have been around the Elixir ecosystem for a while, you have probably heard about “ETS”.

ETS stands for Erlang Term Storage. It’s a storage engine that is built into OTP, and is therefore available in Elixir through the interoperability with Erlang.

ETS is a key / value store, and so you can think of it as pretty much the same as Redis. But because ETS is built right into OTP, you get it for free, without having to rely on a third-party dependancy!

In today’s tutorial we are going be looking at using ETS in Elixir.

What is ETS?

ETS is an in-memory key / value store that is part of OTP and Erlang. ETS stands for Erlang Term Storage and so you can store any Erlang term in ETS without having to serialise and deserialise into a different format.

ETS allows you to store a large amount of data with constant time data access.

As with just about everything else in Elixir and Erlang, each ETS table is created and owned by a process. When the process terminates, the table is automatically destroyed.

Creating tables

ETS is available in Elixir through the interoperability with Erlang. This means we need to use :ets to interact with ETS in Elixir.

To create a new table you use the new/2 function that accepts the name of the table as an atom and a list of options as the second parameter:

table = :ets.new(:todos, [:set, :private])

The first item in the keyword list in the example above is the table type. There are a couple of different types of table you can use in ETS depending on your use case.

In this case I’ve selected :set. This is the default table type, and it is probably what you would most likely expect from a “database” storage mechanism. This table type allows one value per key, and the key must be unique.

You could also use :ordered_set, which, as you would probably be able to guess, is just like a set but ordered by the term.

Next there is :bag which allows many objects per key, but only one instance of each object per key. And finally there is :duplicate_bag, which is the same as :bag but duplicates are allowed.

The second item in the options keyword list is the access control for the table. In the example above I’ve set the table to be :private. This means only the process that created the table can read or write to the table.

You could also have :public, which means any process can read and write to the table. Or finally, you could choose :protected, which means any process can read, but only the owner process can write to the table.

Adding data

Adding data to an ETS table is really easy because it is just a tuple where the first element is the key and the second element is the value:

:ets.insert(table, {:shopping, ["milk", "bread", "cheese"]})

In this case the key is the atom :shopping, and the term is the list.

Reading data

There are a couple of different ways you can read data from an ETS table.

The easiest and quickest method is to search by the key:

:ets.lookup(table, :shopping)

If you are using ETS as a key value store, then this should work really well, even for bigger data sets.

You can also perform matches and more advanced queries against the table to retrieve data from partial matches. However, that is a topic in of itself and I won’t be going into it today. You are best off looking at the ETS documentation.

Deleting data and destroying tables

Deleting an item from the table is really easy, all you need to do is pass the key of the item to the delete/2 function:

:ets.delete(table, :shopping)

If you try to read that item again you will find it’s no longer there.

If you want to also delete the table you can use the delete/1 function:

:ets.delete(table)

Alternatively, if you kill the process, the table will also be taken with it.

What is DETS?

If you are hanging around the Elixir / Erlang community for long enough you will probably also hear about DETS. DETS is similar to ETS, but instead of storing the data in-memory, the data is stored to disk.

DETS also has a similar API to ETS, but instead of using new/2 to create a table, you use open_file/2 instead:

{:ok, table} = :dets.open_file(:shopping, type: :set)

With the table created, using DETS is pretty similar to ETS:

# Insert a row
:dets.insert(table, {:shopping, ["milk", "bread", "cheese"]})

# Find a row by it's id
:dets.lookup(table, :shopping)

If you run the code above in iex and then exit the session you should find a new shopping file in the same directly proving that the data was stored to disk.

Using ETS with GenServer

Hopefully that was a good introduction to using ETS. Let’s now take a look at a practical example of using ETS in combination with GenServer (Understanding GenServer in Elixir). We’re going to create a todo server that can handle many different todo lists.

First up create a new file called todos.ex and add GenServer:

defmodule Todos do
  use GenServer

  def start_link do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
  end
end

As you can see, this is just your standard GenServer implementation. I’m giving the server a name so it’s easier to work with for this example, but in reality you would probably have a todos server for each user.

In the init function we can create a new ETS table for this process and then store it as the state:

def init(:ok) do
  {:ok, :ets.new(:todos, [:named_table, :protected])}
end

By default the table will be :set so we don’t need to pass that option. I’m using the :named_table option so I can access the table by it’s name.

And I’m using the :protected access control so I can read the data from any process, but only the owner process can write to the table.

The first function I will add will be to retrieve a todo list from the server:

def find(name) do
  case :ets.lookup(:todos, name) do
    [{^name, items}] -> {:ok, items}
    [] -> :error
  end
end

This function accepts name of the list as an argument and then attempts to retrieve it from ETS.

I’m using a case statement to pattern match against the result. If the list is found in the table I will return the items, but if the list is not found I will return the atom :error.

The ^ character pins the value. So if the list variable contained the atom :shopping, the pattern match would match against :shopping.

We can take the server for a quick spin to make sure everything is working correctly. Fire up iex and then run the following:

# Start the server
Todos.start_link()

# Try and find the shopping list
Todos.find(:shopping)
# :error

As you can see, we got the :error atom because the shopping list does not exist yet. Let’s add a function to create a new list

First up we can add the public API for creating a new list:

def new(list) do
  GenServer.call(__MODULE__, {:new, list})
end

As you can see, this is just a simple call to the process using GenServer.

On the server side we can handle this request with a handle_call/3 function:

def handle_call({:new, list}, _from, table) do
  case find(list) do
    {:ok, list} ->
      {:reply, list, table}

    :error ->
      :ets.insert(table, {list, []})
      {:reply, [], table}
  end
end

First I attempt to find the table using the find/1 function from earlier. If the list already exists I will just return it from the function, but if the list does not exist I will create it with an empty list and then return the empty list to the client.

Next up we can add a function to add an item to a list, the public API looks like this:

def add(name, item) do
  GenServer.call(__MODULE__, {:add, name, item})
end

In this function I’m accepting the name of the list and the item to add to it.

On the server side we can accept this call with another handle_call/3 function:

def handle_call({:add, name, item}, _from, table) do
  case find(name) do
    {:ok, items} ->
      items = [item | items]
      :ets.insert(table, {name, items})
      {:reply, items, table}

    :error ->
      {:reply, {:error, :list_not_found}, table}
  end
end

Once again I use the find/1 function to find the list. If the list exists I can add the new item to the list of items and then insert the new list into ETS. Finally I will return the new list to the client.

If the list is not found I will return a tuple containing :error and the reason why the error occurred.

Here is this module in full:

defmodule Todos do
  use GenServer

  def start_link do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def init(:ok) do
    {:ok, :ets.new(:todos, [:named_table, :public])}
  end

  def find(name) do
    case :ets.lookup(:todos, name) do
      [{^name, items}] -> {:ok, items}
      [] -> :error
    end
  end

  def new(name) do
    GenServer.call(__MODULE__, {:new, name})
  end

  def add(name, item) do
    GenServer.call(__MODULE__, {:add, name, item})
  end

  def handle_call({:new, name}, _from, table) do
    case find(name) do
      {:ok, name} ->
        {:reply, name, table}

      :error ->
        :ets.insert(table, {name, []})
        {:reply, [], table}
    end
  end

  def handle_call({:add, name, item}, _from, table) do
    case find(name) do
      {:ok, items} ->
        items = [item | items]
        :ets.insert(table, {name, items})
        {:reply, items, table}

      :error ->
        {:reply, {:error, :list_not_found}, table}
    end
  end
end

Now that we have everything in place we can take it for a proper spin. Fire up iex and include the file name to load it into the session:

# Start the todos server
Todos.start_link()

# Create a new todo list
Todos.new(:shopping)

# Add some items to the todo list
Todos.add(:shopping, "milk")
Todos.add(:shopping, "bread")
Todos.add(:shopping, "cheese")

# Find the todo list
Todos.find(:shopping)

# Try and find a todo list that does not exist
Todos.find(:work)
# :error

# Try to add an item to a list that does not exist
Todos.add(:work, "do stuff")
# {:error, :list_not_found}

Conclusion

In today’s tutorial we have looked at using ETS (and DETS) in Elixir.

ETS is an in-memory storage solution that ships with OTP. This means you get it for free when using Elixir because Elixir is built on the foundation of Erlang.

ETS allows you to store any Elixir or Erlang term, it offers very quick lookup, and it can store a lot of data in a couple of different ways.

ETS tables are created by processes. A process can control who can access the data and the ETS is destroyed when the process is killed.

As you can probably imagine, there are many practical uses for ETS as a storage mechanism. I tend to use it in cases where I would have normally of reached for Redis. I love the fact that you a Redis alternative for free just because we are building on the foundation or Erlang an OTP.

Philip Brown

@philipbrown

© Yellow Flag Ltd 2024.