cult3

How to design and build a PHP data export service

Nov 02, 2015

Table of contents:

  1. Designing from the outside in
  2. Creating the Generators
  3. Creating the Writers
  4. Creating the Exporters
  5. Creating the public API
  6. Using the service
  7. Conclusion

Last week we looked at Generating PDFs from HTML and PhantomJS. This is a really nice and simple way to generate PDFs within a web application.

Another common requirement of web applications is the ability to export data as CSV. SaaS applications, particular B2B, well often require this kind of functionality as companies like to have a copy of their data in a easy to use format.

Instead of having separate one-off services for exporting as PDF, or exporting as CSV, we can combine them into a single API. This makes managing this functionality easier and more consistent, and it should promote good practices for adding additional export types in the future if the requirements of the application evolve (as they inevitably will do!).

In today’s tutorial I’m going to walk you through how I would approach building this type of service in a web application, from the initial API design, to the finished product.

Designing from the outside in

When taking on this kind of functionality, I often like to begin my design from the outside and work my way in.

By this I mean, I will design the API of the service, before I worry too much about how it is going to be implemented.

I find this a good approach for a couple of reasons.

First, by thinking about how the service will be used, you get a broad overview of all of the things to consider. There’s nothing worse than being knee-deep in the details, only to realise it’s not going to work because you didn’t consider the bigger picture.

Secondly, it forces you to think of the practicalities of the design, how you can compose the individual components and how you can manage the tricky bits that you will need to give extra consideration to.

And finally, we’re building a service to make our lives easier. One of the most important things to consider is how the design of the API will make this service class easy to use for the consumer.

When exporting data, I want to be able to specify

  1. the “type” of the export (user, project, task)
  2. The “format” of the export (pdf, csv, etc)
  3. any query details that are required to pull the correct data

Therefore the API will probably look something like this:

Export::get($type, $format, $query);

A further consideration is how do I deal with the export once it has been created.

Sometimes I will want to simply return the file as an HTTP response to the request. But I might also want to save the file and then send an email to the user.

This is something else we need to consider, but we can think about that later.

Creating the Generators

As we saw in last week’s tutorial, we can generate a PDF from an HTML template using Laravel’s native View functionality.

In order to add other formats, we need to create similar generator processes that can take a template and data and then transform it into the appropriate format.

To do that, first I will create an interface:

interface Generator
{
    /**
     * Generate the file
     *
     * @param string $template
     * @param array $data
     * @return Writer
     */
    public function generate($template, array $data);
}

Each Generator implementation should be interchangeable and so we can define that contract using this interface. As you can see, we will require a single generate($template array $data) method that will accept a template and an array of data.

Next I will create a CSV implementation. In order to actually create the CSV files, I will be using the league/csv package:

$ composer require league/csv

First I will create the CSVGenerator class and inject an instance of Writer:

use League\Csv\Writer;

class CSVGenerator implements Generator
{
    /**
     * @var Writer
     */
    private $writer;

    /**
     * @param Writer $writer
     * @return void
     */
    public function __construct(Writer $writer)
    {
        $this->writer = $writer;
    }
}

Next I will define a build() method that will do the heavy lifting of generating the template:

/**
 * Build the template data
 *
 * @param string $template
 * @param array $data
 * @return array
 */
public function build($template, array $data)
{
    return call_user_func(require $template, $data);
}

The template in this example is simply going to be a Closure. This is an important step because we’re probably going to have to format the data, select only the relevant data, or do some other kind of transformation.

And finally I will implement the generate() method to write the data to a CSV:

/**
 * Generate the CSV file
 *
 * @param string $template
 * @param array $data
 * @return Writer
 */
public function generate($template, array $data)
{
    return $this->writer->insertAll($this->build($template, $data));
}

I’ve separated the process into two different methods so it will be easier to test the two components in isolation. I want to ensure that the build process is working without having to write the CSV in each test.

Here is an example of a CSV template:

return function ($data) {
    $headers = ["Username", "Email", "Created At"];

    $rows = array_map(function ($row) {
        return [$row["username"], $row["email"], $row["created_at"]];
    }, $data);

    return [$headers, $rows];
};

As you can see, I’m basically just building up an array. This means you can separate the process of creating the array from the model, so you are not tied to just using the toArray() method.

Here are the tests:

class CSVGeneratorTest extends \TestCase
{
    /** @test */
    public function should_build_data_payload()
    {
        $writer = Writer::createFromFileObject(new SplTempFileObject());

        $generator = new CSVGenerator($writer);

        $payload = $generator->build(
            sprintf("%s/closure-template.php", __DIR__),
            [1, 2, 3]
        );

        $this->assertEquals([1, 4, 9], $payload);
    }

    /** @test */
    public function should_generate_csv_payload()
    {
        $writer = Writer::createFromFileObject(new SplTempFileObject());

        $generator = new CSVGenerator($writer);

        $payload = $generator->generate(
            sprintf("%s/closure-template.php", __DIR__),
            [1, 2, 3]
        );

        $this->assertInstanceOf(Writer::class, $payload);
    }
}

In the first test I’m asserting that the build process is correctly transforming the array and returning a new array of data.

In the second test I’m asserting that I’m returned an instance of the Writer class.

You could also make a Generator implementation to wrap the View functionality of Laravel, but I’ll leave that up to you.

Creating the Writers

Next we need to encapsulate the process of writing the data as a specific format.

Once again I will first define an interface:

interface Writer
{
    /**
     * Write the data to a file and return the path
     *
     * @param string $template
     * @param array $data
     * @return string
     */
    public function write($template, array $data);
}

Each Writer implementation will require a write() method.

Next we can create the PDFWriter implementation:

class PDFWriter implements Writer
{
    /**
     * @var Filesystem
     */
    private $files;

    /**
     * @param Filesystem $files
     * @return void
     */
    public function __construct(Filesystem $files)
    {
        $this->files = $files;
    }
}

I’m injecting an instance of a Laravel’s Filesystem so I can write the generated contents to the filesystem.

Next we can define the write() method:

/**
 * Write the data to a file and return the path
 *
 * @param string $template
 * @param array $data
 * @return string
 */
public function write($template, array $data)
{

}

First I will generate the filename of the export. In this case I don’t really care what the filename is so I’m just using a timestamp:

$filename = sprintf("pdf-export-%s.pdf", time());

Next I need to grab the location of the template. Instead of having the PDF templates in the usual resources/views location of Laravel, I have moved them into the Exporting namespace to sit with the rest of these classes.

So in order to get the template, I can use a simple helper function (you don’t need to worry about this):

$template = export("pdf", sprintf("%s.blade.php", strtolower($template)));

Next I will generate the HTML and save it to the filesystem:

$this->storage->put(
    $filename,
    view()
        ->file($template, $data)
        ->render()
);

Again, as I’m not using the default location for the template I will need to use a slightly different syntax for generating the HTML.

Finally we can generate the PDF as we did in last week’s tutorial and return the path to the file:

$path = storage_path(sprintf("app/%s", $filename));

$phantom = base_path("bin/phantom/phantomjs");
$config = base_path("bin/phantom/config.js");
$command = sprintf("%s —ssl-protocol=any %s %s", $phantom, $config, $path);
$process = (new Process($command, __DIR__))->setTimeout(10)->mustRun();

return $path;

The CSVWriter is basically the same:

/**
 * Write the data to a file and return the path
 *
 * @param string $template
 * @param array $data
 * @return string
 */
public function write($template, array $data)
{
    $filename = sprintf('export-%s.csv', time());

    $template = export('csv', sprintf('%s.php', strtolower($template)));

    $this->storage->put($filename, csv()->generate($template, $data));

    $path = storage_path(sprintf('app/%s', $filename));

    return $path;
}

Creating the Exporters

Next we need a way to select the data to export as well as deal with that pesky conundrum of whether to return the file as a HTTP response or simply as a path that can be used to send the file as an email.

The first thing I’m going to do is to create an abstract Exporter class:

abstract class Exporter
{
}

Every class that extends this class will need a build() method to gather the data, so we can define that here:

/**
 * Build the data payload
 *
 * @return array
 */
abstract public function build();

Next we need a method for writing the data:

/**
 * Write the data to a file and return the path
 *
 * @param array $data
 * @return string
 */
public function write(array $data)
{
    $template = head(explode('_', snake_case(class_basename($this))));

    return $this->writer->write($template, $data);
}

In this method I’m getting the name of the template from the child class name and then passing that string and the data to the Writer object.

The method I’m using to get the template name is a bit rough and ready. I had a better way of doing this, but I can’t remember how I did it now.

I will also need a response() method for returning the export as a response. This means we don’t have to deal with this in the controller:

/**
 * Return the Export as a HTTP Response
 *
 * @param string $path
 * @param bool $downloadable
 * @return Response
 */
public function response($path, $downloadable = false)
{
    if ($downloadable) return response()->download($path, basename($path));

    $file = $this->storage->get(basename($path));
    $mime = finfo_file(finfo_open(FILEINFO_MIME_TYPE), $path);

    return response($file)->header('Content-Type', $mime);
}

As you can see, I’m allowing the developer the opportunity to force the response to be a download.

Otherwise I simply get the mime-type of the file and return a response with the correct headers.

Arguably this is not the responsibility of the class, but it will make my life easier so I’m fine with it.

Next you can define Exporter implementations for all of the types of data that can be exported from your application:

For example, here is a ProjectExporter:

class ProjectExporter extends Exporter
{
    /**
     * @var Writer
     */
    protected $writer;

    /**
     * @var array
     */
    private $query;

    /**
     * @param Writer $writer
     * @param array $query
     * @return void
     */
    public function __construct(Writer $writer, array $query)
    {
        $this->writer = $writer;
        $this->query = $query;
    }

    /**
     * Build the data payload
     *
     * @return array
     */
    public function build()
    {
        // Find the project from the query
        $project = Project::find($this->query["id"]);

        // Check the user has permission to the project, yada yada
        // ...

        // Return the Project object in an array
        return compact("project");
    }
}

First I inject an object that implements the Writer interface from earlier, as well as the query options that will be used to find the Project(s) to export.

Next I can implement the build() method to find the project and return it as keyed array.

Creating the public API

Finally we have come full circle and so we can define the public API that we sketched out right at the very start of this article.

I’m going to create a Manager class to encapsulate this process:

class Manager
{
    /**
     * @var Filesystem
     */
    private $files;

    /**
     * @param Filesystem $files
     * @return void
     */
    public function __construct(Filesystem $files)
    {
        $this->files = $files;
    }
}

First I will register all of the Exporters and Writers in the class. You could also inject these through the construct method when registering the class in the IoC:

/**
 * @var array
 */
private $exporters = [
    'project' => ProjectExporter::class,
    'user' => UserExporter::class,
    'account' => AccountExporter::class
];

/**
 * @var array
 */
private $writers = [
    'pdf' => PDFWriter::class,
    'csv' => CSVWriter::class
];

Next I will define two methods to find the Exporter and the Writer. If the Exporter or the Writer do not exist, an Exception will be thrown. This Exception will bubble up to the surface and automatically return the correct HTTP response as we saw in Dealing with Exceptions in a Laravel API application:

/**
 * Get the Exporter
 *
 * @param string $name
 * @return string
 */
private function exporters($name)
{
    if (array_has($this->exporters, $name)) {
        return array_get($this->exporters, $name);
    }

    throw new InvalidExportType('invalid_export_type', [$name]);
}

/**
 * Get the Writer
 *
 * @param string $name
 * @return string
 */
private function writers($name)
{
    if (array_has($this->writers, $name)) {
        return array_get($this->writers, $name);
    }

    throw new InvalidExportFormat('invalid_export_format', [$name]);
}

Next I will create the public get() method that will accept the type, format and query parameters:

/**
 * Get an Exporter
 *
 * @param string $exporter
 * @param string $format
 * @param array $query
 * @return Exporter
 */
public function get($exporter, $writer, array $query)
{
    $exporter = $this->exporters($exporter);
    $writer = $this->writers($writer);

    return new $exporter(new $writer($this->files), $query);
}

Finally I will define a Facade to make this service feel more at home in a Laravel application:

use Illuminate\Support\Facades\Facade;

class Export extends Facade
{
    /**
     * Get the registered name of the component
     *
     * @return string
     */
    protected static function getFacadeAccessor()
    {
        return "export";
    }
}

Using the service

Generating an export from a Controller is now really easy:

/**
 * Create an Export
 *
 * @param Request $request
 * @return Response
 */
public function export(Request $request)
{
    $format = $request->input('format');
    $download = $request->input('download');
    $query = $request->except(['type', 'format', 'download']);
    $exporter = Export::get($type, $format, $query);

    return $exporter->response($exporter->write($exporter->build), $download);
}

This will automatically create the right export type, gather the required data and return the correct HTTP response.

Alternatively, you could use the same process but miss out the final call to the response method to generate the export and then send it as an email:

$format = $request->input("format");
$query = $request->except(["type", "format"]);
$exporter = Export::get($type, $format, $query);
$path = $exporter->write($exporter->build());

// Send the email

Conclusion

In today’s tutorial we looked at how to build a service for exporting data from a Laravel application. This involved designing the API and thinking about how the individual components could be composed together.

We can provide two common formats, but it won’t be difficult to add additional formats in the future.

We have separate the various components of this overall service into individual responsibilities that can be tested in isolation.

We have provided a way to customise how the data appears in the PDF or CSV via templates.

And we’ve provided a way to gather the data and enforce any business rules whenever an export is requested.

Even if you are not going to provide exporting as an option in your application, I hope you have picked up a couple of techniques that will make designing and building your internal services a lot easier!

Philip Brown

@philipbrown

© Yellow Flag Ltd 2024.