cult3

Importing demo data into a Laravel application

Jan 11, 2016

Table of contents:

  1. How this is going to work
  2. How do we store the data?
  3. Reading the data
  4. Creating the Model objects
  5. Mapping the data
  6. Importing the data
  7. Creating the Manager class
  8. Conclusion

The on-boarding experience that your users experience when they are first getting to grips with your application will make or break you.

Trying a new application is time consuming and will require a lot of investment and so your potential customers need to immediately understand exactly how you will solve their problems.

A lot of applications will sign up a new user and then dump them into a blank state. This puts all of the work onto the user to figure out how to use the application, and understand if it will work for them.

A much better approach is to put the user into a demo account to see how the application works in full flow.

This sandbox account will allow the user to experiment, learn the tools, and better appreciate your solution for their problem.

In today’s tutorial we will look at importing demo data to create better on-boarding flows for your Laravel application.

How this is going to work

Whenever a new user signs up to the application, we need to create a demo account for the user to play with.

This will entirely depend on the context of your application, but lets just pretend we’re building a project management application. In this case we will need a demo project.

To make this a realistic project, we need to pre-populate it with realistic data.

For example, we will need a selection of tasks that are completed, in progress, and still left to be started.

We will need discussion messages, forum posts, and comments. Dates in the calendar, milestones, and probably a whole load more things to put the user into the middle of a realistic project situation.

So today we’re going to build a way of managing all of this data and importing it into the application.

We will also need a way of allowing multiple different demo contexts.

For example, if our project management application is used by teachers, government officials, or application developers, we should be providing a demo that is relevant to their industry or role.

So let’s get started!

How do we store the data?

The first thing we need to consider is “how do we store the data?”.

As I mentioned in the previous section, we need to set up a project that feels real. This will involve a lot of intricately related data.

This data will also evolve over time, especially as we introduce new features into the product.

The simple solution is to just have a script that runs through and creates the data in the database.

But this is brittle and it means whenever someone in the company wants to tweak the data, they have to go through you.

My preferred option is to use Excel spreadsheets. An Excel spreadsheet is an easy to use container for any type of data. And even non-technical people will have no problem modifying the data in a spreadsheet.

We can represent each different “resource” on different worksheets to keep things neatly organised. And we can keep each “industry” as a separate spreadsheet.

At the end of the day, how you store the data doesn’t really matter, but I’ve found using Excel spreadsheets to be a really good option.

Reading the data

Next we need to create the class to read the data from the source.

As I mentioned above, it doesn’t really make that much of a difference where you store the data. But for this tutorial I’m going to be using Excel Spreadsheets as my source.

And potentially we should be able to switch to a different source at some point in the future without any disruption.

So first we need to define an interface:

interface Reader
{
    /**
     * Read the data from the source
     *
     * @return StdClass
     */
    public function read();
}

As you can see, we only need a single read() method.

Next I will create a the ExcelReader implementation:

class ExcelReader implements Reader
{
    /**
     * @var PHPExcel
     */
    private $spreadsheet;

    /**
     * @param PHPExcel $spreadsheet
     * @return void
     */
    public function __construct(PHPExcel $spreadsheet)
    {
        $this->spreadsheet = $spreadsheet;
    }

    /**
     * Read the data from the source
     *
     * @return StdClass
     */
    public function read()
    {
    }
}

Here I’m injecting an instance of PHPExcel which will be the Excel spreadsheet that we loaded from the filesystem (or wherever you store your data source).

Next we need to convert the spreadsheet into an object of data attributes that we can use to import.

To do this I’m going to create a new StdClass and then add properties for each of the resources of the application as instances of Laravel’s Collection class:

$data = new StdClass();
$data->tasks = collect();
$data->messages = collect();
$data->todos = collect();
$data->milestones = collect();

Next we need to transform the raw arrays of data into objects that represent each type.

This means we can deal with an validations or transformations of that particular type of data within the context of it’s own object.

First we need to iterate through each of the worksheets to get the data:

foreach ($this->spreadsheet->getAllSheets() as $sheet) {
}

Next we can get the class name for the worksheet:

$title = $sheet->getTitle();
$model = $this->getModelNamespace($title);

In getModelNamespace you basically just need to form the correct namespace for the title of the worksheet, and then ensure that the class actually does exist:

/**
 * Get the Model namespace
 *
 * @param string $name
 * @return string
 */
private static function getModelNamespace($name)
{
    $class = ucfirst(str_singular($name));

    $class = sprintf('your-namespace\%s', $class);

    if (class_exists($class)) return $class;

    throw new InvalidImportModel(
    sprintf('%s is an an invalid worksheet name', $name));
}

Next we can convert the worksheet into an array and get the headers:

$rows = $sheet->toArray();
$headers = array_shift($rows);

Next we can iterate over the rows, combine each row with the headers array, and then covert them into model objects:

$collection = collect(
    array_filter(
        array_map(function ($row) use ($headers) {
            if (!array_filter($row)) {
                return;
            }

            return array_combine($headers, $row);
        }, $rows)
    )
)->map(function ($row) use ($model) {
    return $model::fill($row);
});

$data->$title = $data->$title->merge($collection);

Finally, once each worksheet has been iterated over, you can return the $data object from the method.

The test for this class looks like this:

class ExcelReaderTest extends \TestCase
{
    /** @test */
    public function should_read_spreadsheet()
    {
        $spreadsheet = base_path("app/Templates/data/default.xlsx");

        $reader = new ExcelReader(PHPExcel_IOFactory::load($spreadsheet));

        $this->assertInstanceOf("StdClass", $reader->read());
    }
}

Here I’m passing a default Excel spreadsheet that I’ve created and then I’m simply asserting that the return value is the StdClass I was expecting.

Creating the Model objects

Next we need to create the Model objects that will be used to validate the incoming data from the template.

These are basically just plain PHP objects that accept the required parameters and perhaps do a little bit of validation to ensure that the data is in the correct format.

How you do this is really up to you and what is important about the data you are importing.

Realistically the data in the spreadsheets is not going to change very often. When the data does change, you only need to test the process once to ensure it works, so I wouldn’t go nuts with validation logic in these simple PHP classes.

Mapping the data

As data is imported into the application, each model will be updated with the auto incrementing id from the database.

Before we import the related data, we need to map these ids to the remaining models.

For example, if the first resource we imported was projects, we would need to map the auto incrementing project ids to the tasks so that we know what task is related to what project.

To do that, we can create a mapper class that can hold this process.

First we need to create an interface:

interface Mapper
{
    /**
     * Map the data
     *
     * @param StdClass $data
     * @return StdClass
     */
    public function map(StdClass $data);
}

The Mapper classes will accept the full data payload and then return it again so the data can just flow through each class.

Next we can create the first implementation:

class TaskMapper implements Mapper
{
    /**
     * Map the data
     *
     * @param StdClass $data
     * @return StdClass
     */
    public function map(StdClass $data)
    {
        $data->tasks = $data->tasks->map(function ($record) use ($data) {
        // Map the tasks to the projects
        });

        return $data;
    }
}

This process is really dependant on the data you are importing and so it’s hard to show an example that doesn’t confuse things.

Basically you just need to get the project index from the task, get the corresponding project, make sure the project does exist, and then assign the auto-incrementing id to the task.

Importing the data

Finally we can create the classes to actually import the data.

First we will define an interface:

interface Importer
{
    /**
     * Run the import process
     *
     * @param StdClass $data
     * @return StdClass
     */
    public function import(StdClass $data);
}

Again the implementation should accept the StdClass of data and then return it so that the importers and mappers can flow sequentially.

Next we can define the importer classes:

class TaskImporter implements Importer
{
    /**
     * Run the import process
     *
     * @param StdClass $data
     * @return StdClass
     */
    public function import(StdClass $data)
    {
        // Import the data
    }
}

Each Importer implementation will contain the logic of building the model objects and its relations from the StdClass of data.

Creating the Manager class

With everything in place we can now define a Manager class to slap a public API on this process.

class Manager
{
}

The public API of this class will be a single import method that will accept the name of the template to import:

/**
 * Import a template
 *
 * @param string $template
 * @param bool $with_data
 * @return void
 */
public function import($template)
{

}

First we will grab the path to the template:

/**
 * Get the path for the template
 *
 * @param string $template
 * @return string
 */
public function path($template)
{
    $path = base_path(sprintf('app/Templates/data/%s.xlsx', $template));

    if (file_exists($path)) return $path;

    throw new TemplateNotFound('template_not_found', [$template]);
}

If the template is not found we can just bail here with an Exception.

Next we can read the template:

/**
 * Read the template
 *
 * @return PHP_Excel
 */
public function read($path)
{
    return (new ExcelReader(PHPExcel_IOFactory::load($path)))->read();
}

Alternatively you might want to inject the Reader implementation into the class, but I’m not going to change the implementation so I’m fine with this approach.

Finally, you can run through each of your importer and mapper implementations to import the data:

/**
 * Import a template
 *
 * @param string $template
 * @param bool $with_data
 * @return void
 */
public function import($template)
{
    $path = $this->path($template);
    $data = $this->read($path);

    $data = (new ProjectImporter)->import($data);
    $data = (new TaskMapper)->map($data);
    $data = (new TaskImporter)->import($data);
}

And of course, to give the service that Laravel feeling, we can add a Facade:

class Template extends Facade
{
    /**
     * Get the registered name of the component
     *
     * @return string
     */
    protected static function getFacadeAccessor()
    {
        return "template";
    }
}

Conclusion

A good on-boarding process is critical to the success of your application. Dropping the user off into a blank canvas will be terrible for your conversion rates.

You shouldn’t expect the user to intuitively understand how to use your product and how it will solve their problems.

Dropping the user into the middle of a working “project” is a much better solution. This gives the user an immediate understanding of how the application will work.

The user can get to grips with the tools and terminology of your application within the safety of a sandbox. And they will be far more likely to understand the benefits of using your application and becoming a paying customer.

The approach I’ve shown you today allows you to define custom industry specific demos for your product. This is great if you have built a product that spans multiple industries.

It also empowers the non-technical members of your team to tweak the data based upon customer feedback without having to ask you to make every change for them.

This ensures the demo data is tidy and well organised, it’s easy to add new industry demos, and it’s easy to evolve the data over time as your application grows!

The implementation details of today’s tutorial has admittedly been a little bit vague. However, in this situation I don’t think you need to see every line of code of an imaginary implementation.

The real take away is the theory, not the implementation, but if you do have any specific questions, leave a comment and I’ll be happy to answer!

Philip Brown

@philipbrown

© Yellow Flag Ltd 2024.