2. Tutorial

Juniper comes with a set of examples meant to show a developer working in the serverless domain, what he/she needs to do in order to easily adopt the tool.

To get started, make sure you clone the project and install juni:

>>> git clone git@github.com:eabglobal/juniper.git
>>> cd juniper
>>> python3 -m venv venv
>>> source venv/bin/activate
>>> pip install -e .
>>> cd examples/processing

2.1. Map Reduce Pipeline

The processing example is based on a map reduce pipeline in which the developer has defined multiple lambda functions. The first lambda function in the processing pipeline is the split_lambda. This function will be triggered whenever a json object is added to S3, the function will read the file and it will split it into two files.

The entire business logic associated with that function is:

import json
from commonlib import s3
from splitter.constants import BUCKET_NAME


def lambda_handler(event, context):
    """
    Reads a source file from s3, splits the contents of the file in two and
    adds the resulting values in the same bucket.
    """

    file_name = event.get('source_file')

    data = s3.read_json_file(BUCKET_NAME, file_name)
    split_point = len(data) // 2

    s3.write_json_file(BUCKET_NAME, 'to_map/part1.json', data[:split_point])
    s3.write_json_file(BUCKET_NAME, 'to_map/part2.json', data[split_point:])

    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Created 2 files'})
    }

Note that this function has a reference to the commonlib.s3 object. The commonlib module lives along side the split_lambda function and it must be included in the split_lambda.zip artifact. Otherwise, the function will fail with a reference not found error.

The business logic of the remaining functions in the pipeline is not relevant to the example, however, the folder structure has a set of placeholders for them.

The manifest file, for the generation of the artifacts of this pipeline looks like:

functions:
  split_step:
    requirements: ./requirements.txt
    include:
      - ./commonlib
      - ./split_lambda/splitter

  map_step:
    requirements: ./requirements.txt
    include:
      - ./commonlib
      - ./map_lambda/mapper

  reduce_step:
    image: lambci/lambda:build-python3.7
    requirements: ./reduce_lambda/requirements.txt
    include:
      - ./reduce_lambda/reducer/lambda_function.py

Note that in the definition of the split_step the includes section makes reference to the commonlib. This line indicates to juniper that the codebase inside the commonlib folder must be included in the .zip artifact for the split function.

>>> juni build
Removing processing_map_step-lambda_1    ... done
Removing processing_split_step-lambda_1  ... done
Removing processing_reduce_step-lambda_1 ... done
Removing network processing_default
Creating network "processing_default" with the default driver
Creating processing_split_step-lambda_1  ... done
Creating processing_map_step-lambda_1    ... done
Creating processing_reduce_step-lambda_1 ... done
Attaching to processing_split_step-lambda_1, processing_map_step-lambda_1, processing_reduce_step-lambda_1
split_step-lambda_1   | Starting to package split_step
reduce_step-lambda_1  | Starting to package reduce_step
map_step-lambda_1     | Starting to package map_step
reduce_step-lambda_1  | Finished packaging
processing_reduce_step-lambda_1 exited with code 0
split_step-lambda_1   | Finished packaging
processing_split_step-lambda_1 exited with code 0
map_step-lambda_1     | Finished packaging
processing_map_step-lambda_1 exited with code 0

Without any arguments, juni will by default look for a file named manifest.yml. From that file, juni will create the artifacts in a ./dist folder. The output of the above command creates the following artifacts:

>>> tree dist
dist
├── map_step.zip
├── reduce_step.zip
└── split_step.zip

Unzipping the split_step lambda we expect to have the contents of pip installing the request library; given that it is defined in the requirements.txt. The contents of the common lib and finally the splitter. All of which are defined in the manifest file.

>>> ls -larth
drwx------  10 pdiazvargas  375838832   320B Jan 30 22:27 .
drwxr-xr-x   7 pdiazvargas  375838832   224B Jan 30 22:27 ..
drwxr-xr-x  15 pdiazvargas  375838832   480B Jan 31  2019 urllib3
drwxr-xr-x   4 pdiazvargas  375838832   128B Jan 31  2019 splitter
drwxr-xr-x  20 pdiazvargas  375838832   640B Jan 31  2019 requests
drwxr-xr-x  10 pdiazvargas  375838832   320B Jan 31  2019 idna
drwxr-xr-x   4 pdiazvargas  375838832   128B Jan 31  2019 commonlib
drwxr-xr-x  42 pdiazvargas  375838832   1.3K Jan 31  2019 chardet
drwxr-xr-x   6 pdiazvargas  375838832   192B Jan 31  2019 certifi
drwxr-xr-x   3 pdiazvargas  375838832    96B Jan 31  2019 bin