Filter Overview

Filters are customizable blocks of Python code that sit between Sources and Destinations. Filters are used to transform the Payload in a variety of ways, or to make decisions about whether the Payload should continue being processed. For example, if you have an Event with a JSON payload, but you need to consume that Event as XML, you can use a Filter to transform that Payload. If you have a Source that receives many event types, but you are only interested in flowing one of those types on to a Destination, the Filter can be used to stop processing in its branch upon meeting your configured conditions. The Atlas platform also includes a variety of Utility classes for use in Filters which simplify the transformation of payloads.

Filter Basics

Every filter definition starts by defining a method run taking one parameter. For example:

def run(payload):
    # do nothing
    return payload

Inside your run method body, you can manipulate the payload as you see fit. If you return None from your filter, the routing for this message will stop in that branch of the Route, otherwise the payload object you return will be passed to the next Filter in the chain as defined by your Route or to the Destination if it’s the final Filter. See Routes for how fanning out multiple branches works.

The Payload Object

The payload object that is passed into your run method is a Payload-type object which contains the payload data. Its structure differs based on the type of the payload.

HTTP Payloads

For HTTP payloads, the object has the following structure:

payload = Payload()


payload.query = "<this contains the query>"
payload.body = "<this contains the body>"
payload.headers = {
    "header": "values",
    "go": "here"
}

# this is always set for an HTTP Payload
payload.type = PAYLOAD_TYPE_HTTP 

# this is initially empty, see VARS section for how to use
payload.vars = {
    
}

Raw Bytes Payloads

For Raw Bytes payloads, the object has the following structure:

payload = Payload()


payload.body = "<this contains the body>"


# this is always set for an HTTP Payload
payload.type = PAYLOAD_TYPE_RAWBYTES

# this is initially empty, see VARS section for how to use
payload.vars = {
    
}

Filter Creation

In the Web Console, navigate to Resources » Filters. Select the + button to create a new filter.

In the creation modal, give the filter a name, description, and select from one our templates to get you started. When finished, click the Create button and you’ll be taken to the Filter’s viewing page.

Filter Configuration

A filter is configured in the Filter Designer. You can access the Filter Designer from the Filter’s viewing page by clicking the Edit button at the bottom. Check out the documentation for the Filter Designer for more info on how to create/edit/debug/update a filter.

Payload Vars

Every payload has a special variable called vars which can be used to pass info to the eventual destination. For example, if you need to change the URL for and HTTP destination based on some filter input condition, you could set the variable $URL$ in the vars dict.

Vars Example

API_BASE = "https://example.com/api/v1/"

def run(payload):
    if payload.type == PAYLOAD_TYPE_HTTP:
        jobj = json.loads(payload.body.decode())
        
        if jobj.get("api_path", None) is not None:
            payload.vars["$URL$"] = API_BASE + jobj["api_path"]

    return payload

Filter Utilities

A number of utility classes and constants have been added to make Filter design and use simpler.

• Constants
• time
• datetime
• dateutil
• XMLElement
• json
• urllib
• hashlib
• hmac
• Miscellaneous

Constants

The following constants are available for use inside filters:

• PAYLOAD_TYPE_HTTP - the HTTP payload type
• PAYLOAD_TYPE_RAWBYTES- the Raw Bytes payload type

time

The time module has the same API as the standard Python time module.

datetime

The datetime module has the same API as the standard Python datetime module.

Example

def run(payload):
    two_days_ago = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(days=2)
    print(f"Two Days Ago = {two_days_ago}")
    if payload.headers is not None:
        the_date_str = payload.headers.get("Date", None)
        if the_date_str:
            x = dateutil.parser.parse(the_date_str)
            print(x) # stick it in the debug output
            if x < two_days_ago:
                print("Header value was before two days ago")
    return payload

dateutil

The dateutil module has the same API as the popular Python dateutil package.

Example

def run(payload):
    if payload.headers is not None:
        the_date_str = payload.headers.get("Date", None)
        if the_date_str:
            x = dateutil.parser.parse(the_date_str)
            print(x) # stick it in the debug output
    return payload

json

The json module has the same API as the standard python json module.

urllib

The urllib module is a subset of the standard Python urllib module. It pulls in the urllib.parse subset. See urllib.parse documentation.

hashlib

The hashlib module has the same API as the standard Python hashlib module.

hmac

The hmac module has the same API as the standard Python hmac module.

XMLElement

The XMLElement utility class provides XML tree creation and walking facilities.

Methods

# [constructor] returns an XMLElement object
# name:     the XML element's name (what appears between the angle braces)
# value:    the value inside the XML node
# attrs:    a dictionary of key=value pairs of the XML element's attributes (for example {"debug": "on", "level": 4} get's rendered as <element_name debug="on" level="4">element_value</element_name>)
# children: a list of XMLElement objects that are the direct children of this node
XMLElement(name: str, value:str=None, attrs: dict=None, children:list=None) -> XMLElement

# [class method] walks the XML tree and calls your callback for each element. If the return value from your callback is False, the tree walk is stopped.
XMLElement.walk(callback)

# [class method] adds an XMLElement child to the children list of the given element
XMLElement.add_child(child: XMLElement)

# [class method] renders the string representation of the XML tree from this node down. Can also be invoked by str(xml_obj)
XMLElement.__str__() -> str

# [class member] name of this node
XMLElement.name: str

# [class member] value of this node
XMLElement.value: str

# [class member] attributes of this node
XMLElement.attrs: dict

# [class member] children of this node
XMLElement.children: List[XMLElement]

Examples

def run(payload):
    # convert the incoming payload to JSON
    obj = None
    try:
        obj = json.loads(payload.body)
    except:
        return None # couldn't parse

    # create our root element for the XML
    xRoot = XMLElement("root")
    
    # define a recursive capability taking:
    # - o: a dict object
    # - p: the XMLElement parent object
    def recurse(o, p):
        if o is None:
            return
        # iterate the object and recurse if the values
        # are dicts themselves
        for k, v in o.items():
            xChild = XMLElement(k)
            if Type.is_dict(v):
                recurse(v, xChild)
            elif Type.is_list(v):
                # if it's a list, don't recurse, but "list-ize" the values
                # this can be modified to suit the XML syntax for arrays
                # you expect, but as a default it generates numbered elements
                # for example:
                # {"an-array":["cats", "dogs"]} turns into 
                # <an-array>
                # <an-array0>cats</an-array0>
                # <an-array1>dogs</an-array1>
                # </an-array>
                i = 0
                for val in v:
                    xListItem = XMLElement(f"{k}{i}", val)
                    xChild.add_child(xListItem)
                    i = i + 1
            else:
                # if it's just a string, number, or null, just assign it
                xChild.value = str(v)
            # add a child
            p.add_child(xChild)
        
    recurse(obj, xRoot)
    payload.body = str(xRoot)
    return payload

Miscellaneous

The following non-exhaustive list of miscellaneous items are provided as helpers to building Filters.

print

If you need to access debug statements while designing a filter, you can use the standard print call. The output will be provided back when you validate in the Filter Designer.

Basic Types

The basic types in Python (int, float, str, dict, set, list) are all provided.

Classes

Classes can be defined inside the filter and used as part of the logic.