Unix Pipelines with cURL requests and Serverless functions

How you can take advantage of serverless infrastructure to create small command line tools that will help you to manipulate text without leaving the UNIX philosophy

Yes, I’m talking about lambda functions that can be called in a UNIX pipeline by using cURL.

We know that we have AWK, SED, CUT, UNIQ, SORT and other great tools to manipulate text in UNIX, those tools were written more than 30 years ago and nowadays we still find them useful.

Most of the time, those tools are more than enough to do whatever you want, but, what happen when you need a very specific tool and you don’t want to write a very complex sed, perl or awk script?

One thing you can do, is to write your own script in your favorite programming language and connect it to your pipeline, but what if you’re in an environment where that programming language is not available? do you want to spend time reading bash, perl, awk or sed documentation? probably you don’t.

Here is where cURL and Serverless can play very well together.
Having cURL and internet, you have access to a big infrastructure and almost any programming language runtime without installing it on your host.

Let’s check this example:

Given this file (curl -Ss https://www.metaweather.com/api/location/search/?query=Madrid), we want to format with newlines and indentation.

Since we’re assuming we don’t have NodeJS or Python available, only cURL, we’re going to create a Serverless function with Vercel to do the job and return the output to our command line pipeline: