Handling complex API requests with Pipelines
Oct 05, 2015
Table of contents:
API’s have probably never been more important than they are right now. An API is no longer just a method for making interesting mashups, they are now the cornerstone for the entire application architecture.
An API is the interface to your application that enables rich client-side experiences, mobile applications and integrations with third-party partners, and customers.
Your API should be able to handle complex requests for data for the myriad of ways the API could be used.
This can quickly get out of hand if you try and just deal with it in the Controller.
In last week’s tutorial (How to use the Pipeline Design Pattern in Laravel) we looked at the Pipeline Design Pattern and how to use Laravel’s native pipeline functionality.
In today’s tutorial I’m going to show you how I deal with complex queries using pipelines.
The road to ruin
Before we get into the meat and potatoes of using the Pipeline Design Pattern to build complex queries, first we will look at why this abstraction is required in the first place.
It is very easy to build an API using a web framework such as Laravel.
For example, imagine we are building a project management application. We could return all of the projects from the /projects endpoint like this:
public function index()
{
return Project::all();
}
The Collection will automatically be returned as JSON when it is covered into a response. That was pretty easy right?
But if course, we only want to return the projects for the currently scoped account, rather than the projects for all accounts. To solve this we can add a WHERE clause:
public function index()
{
$account = Context::get('Account')->model();
return Project::where('account_id', $account->id)->get();
}(We looked at getting and setting the context in Managing Context in a Laravel application and Setting the Context in a Laravel Application).
Ok, so that’s still pretty easy, right?
Uh oh, we’ve just had a request from the front-end developers. They need the project list to be returned in alphabetical order. Ok, that shouldn’t be too much trouble:
public function index()
{
$account = Context::get('Account')->model();
return Project::where('account_id', $account->id)
->orderBy('name')
->get();
}Hmm, this method is starting to get a bit bigger now, but it’s not too bad.
The front-end developers have come back and would also like to optionally order the projects by different columns, hmm ok, this is getting tricky now:
public function index(Request $request)
{
$account = Context::get('Account')->model();
$query = Project::where('account_id', $account->id);
if ($request->has('sort')) {
$query->orderBy($request->get('sort'));
} else {
$query->orderBy('name');
}
return $query->get();
}The Project Manager has been doing customer interviews, apparently customers want to be able to filter the projects by different columns, hmm this is getting really messy:
public function index(Request $request)
{
$account = Context::get('Account')->model();
$query = Project::where('account_id', $account->id);
if ($request->has('sort')) {
$query->orderBy($request->get('sort'));
} else {
$query->orderBy('name');
}
foreach ($request->except(['sort']) as $key => $value) {
$query->where($key, $value);
}
return $query->get();
}Uh oh, the application is really taking off and some accounts have thousands of projects, we need to limit the number of accounts returned to 10, but allow clients to optionally request up to 100:
public function index(Request $request)
{
$account = Context::get('Account')->model();
if ($request->has('limit') && $request->input('limit') > 100) {
throw new InvalidRequestLimit('invalid_request_limit');
}
$limit = $this->request->input('limit', 10);
$query = Project::where('account_id', $account->id);
if ($request->has('sort')) {
$query->orderBy($request->input('sort'));
} else {
$query->orderBy('name');
}
foreach ($request->except(['sort', 'limit']) as $key => $value) {
$query->where($key, $value);
}
return $query->take($limit)->get();
}As you can see, this Controller method has very quickly got out of hand with only a couple of querying options and without any kind of error checking.
There has to be a better way!
Pipelines to the rescue!
If you remember back to last week, we looked at the Pipeline Design Pattern as a method of breaking up a monolithic process into individual tasks.
This has the benefit of making each task a composable unit that can be added, removed or replaced within the context of a bigger process.
The pipeline accepts data and then passes it through each of the tasks. Each task has a single responsibility and can therefore be tested in isolation.
An excellent use-case for the Pipeline Design Pattern is middleware.
However we can use the same pattern for building up complex queries.
First we pass in the Query Builder object and the request at the start of the pipeline.
Next we pass the Query Builder through each task.
If the request specifies that query option, we can add it to the builder object.
If the request does not specify that query option, we can just pass the Query Builder to the next stage.
If one of the query options is invalid or breaks our business rules, we can throw an exception that will halt the process and return the correct HTTP response, see Dealing with Exceptions in a Laravel API application.
Creating the Query Pipes
So the first thing we need to do is to create the query pipes. Each pipe should have a handle() that accepts the Request and a Closure object.
Here is the pipe to set the Account context:
/**
* Handle the process of the Pipe
*
* @param Request $request
* @param Closure $next
* @return Builder
*/
public function handle(Request $request, Closure $next)
{
$builder = $next($request);
$account = Context::get('Account')->model();
$builder->where('account_id', $account->id);
return $builder;
}Here is the sort pipe:
/**
* Handle the process of the Pipe
*
* @param Request $request
* @param Closure $next
* @return Builder
*/
public function handle(Request $request, Closure $next)
{
$builder = $next($request);
if ($request->has('sort')) {
foreach ($request->input('sort') as $sort) {
$builder->orderBy($sort);
}
}
return $builder;
}And here is the limit pipe:
/**
* Handle the process of the Pipe
*
* @param Request $request
* @param Closure $next
* @return Builder
*/
public function handle(Request $request, Closure $next)
{
if ($request->get('limit', 100) > 100) {
throw new InvalidRequestLimit('invalid_request_limit');
}
return $next($request);
}This pipe is only checking the business rule, it’s not adding anything to the query.
Creating the Pipeline
Using the pipeline is very easy. First we resolve the pipeline from the IoC container or optionally instantiate it yourself:
$pipeline = app("Illuminate\Pipeline\Pipeline");
Next we send in the Request object:
$pipeline->send($request);Next we pass the tasks as an array of pipes:
$pipeline->through([
/* */
]);Next we grab an instance of the Query Builder:
$builder = Project::query();
Finally we can pass the Query Builder to the then() method as the destination callback and accept the Query Builder on the other side:
$builder = $pipeline->then(function ($request) use ($builder) {
return $builder;
});
Now the Builder object is ready to be resolved you can call any of the methods to execute the query such as first(), get() or count().
Conclusion
Dealing with complex queries can be tough! As an API developer, it’s your responsibility to build an application that can handle the demands and requests of the consumers of your API and your customers.
But dealing with that complexity can be difficult if you do not make the right abstractions.
As you can see from the first example, trying to handle all of that complexity and different options as one monolithic process is going to be a nightmare.
It’s also going to be really difficult to add new query parameters or remove old ones from a monolithic process.
By using the Pipeline Design Pattern, you don’t face that problem. Each individual task has a single responsibility and so each task is responsible only for that one particular query parameter.
Each task can be tested in isolation, and it’s very easy to add, remove or replace tasks from the process.
Today’s tutorial has been a simple example of how I deal with this kind of complexity. In my real world projects, my implementation is actually a bit more involved than what I’ve shown you today.
In the not so distant future I want to release a package that will make dealing with this kind of complexity even easier by allowing you to accept all sorts of complex query parameters and a blueprint for easily integrating this functionality into your application. (I haven’t written a single line of this package yet, so don’t expect anything too soon!)
But until then, I hope this example has given you food for thought.
