Building a Generic Rules System for Booster

BoosterKnowledgeNode-jsTutorialsUtilities

Aug 5

Written By KB

During my time at Booster, on several occurrences we've come across the need to perform some operation based on some conditions.

For example: Imagine you've got a promotion that gives $5 off your next order. But you'd only like to grant that promotion after the user places his first order. In that scenario you would have to write some logic in code to check the user's order count perhaps and compare against it. Another example is having a promotion that can only be applied on a specific order with a specific amount.

As you can imagine that it might get a bit cumbersome to have to code all of that in the core business logic for every case the Product Manager comes up with. What if there could be a system that has access to the "user in context" among other things, and based on that write the conditions in a "component based" approach where we can easily change rules without having to redeploy new builds.

Hence the birth of what we call the "Requirement System" at Booster!

Data Model Examples

Before I dive deep into what all the data models and their properties, I'll showcase some examples of the data the system will consume:

"requirement": {
  "operand": "and",
  "nestedRules": [{
    "comment": "Customer has not placed an order since May 5th, 2021",
    "operand": "lessThan",
    "property": {
      "propertyChain": "{{customer.lastFueledAt}}"
    },
    "comparisonValues": [{
      "constant": "2021-05-01T00:00:00.000+00:00"
    }]
  }, {
    "comment": "Customer has ordered before",
    "operand": "greaterThan",
    "property": {
      "propertyChain": "{{customer.numCompletedRequests}}"
    },
    "comparisonValues": [{
      "constant": 0
    }]
  }]
}

In this sample, you can observe a couple of things:

We have 2 rules in the nestedRules property.
The first one checks to see if the customer has placed an order since May 5th by comparing the customer.lastFueledAt with a specific date.
The second condition checks to see if the customer has at least 1 completed order. This is done by comparing the property customer.numCompletedRequests with the constant 0.

Now you might say where do I write this "requirement"? What we've done was store it in mongodb directly as such:

// promotion
{
  "name": "$5 off your next order",
  "numUses": 1,

  "requirement": {
    "operand": "and",
    "nestedRules": [{
      "comment": "Has ordered before",
      "operand": "greaterThan",
      "property": {
        "propertyChain": "{{customer.numCompletedRequests}}"
      },
      "comparisonValues": [{
        "constant": 0
      }]
    }]
  }
}

A slightly more complex example with database queries:

// ...
"requirement": {
  "operand": "and",
  "nestedRules": [{
    "comment": "User is part of this feature's experiment",
    "operand": "has",
    "property": {
      "query": {
        "collection": "CustomerExperiment",
        "method": "findOne",
        "where": {
          "and": {
            "customerId": "{{customer._id}}",
            "experimentId": ObjectId("5ca3e0db3993c0001e69e2a8")
          }
        }
      },
      "propertyChain": "{{result.variantId}}"
    },
    "comparisonValues": [{
      "constant": "B"
    }]
  }, {
    "comment": "User is not part of the booster assistant experiment",
    "operand": "has",
    "property": {
      "query": {
        "collection": "CustomerExperiment",
        "method": "findOne",
        "where": {
          "and": {
            "customerId": "{{customer._id}}",
            "experimentId": ObjectId("5c4279129307cf00077661dc")
          }
        }
      },
      "propertyChain": "{{result.variantId}}"
    },
    "comparisonValues": [{
      "constant": "A"
    }, {
      "constant": "EXCLUDED_FROM_EXPERIMENT"
    }]
  }, {
    "operand": "has",
    "property": {
      "propertyChain": "{{customer.numCompletedRequests}}"
    },
    "comparisonValues": [{
      "constant": 0
    }]
  }]
}

As you can see in the above example:

We first check if the user is part of an Experiment. That is done using a database query on the "CustomerExperiment" collection by providing the "customerId": "{{customer._id}}" as a query parameter, along with the experimentId. Note how we have the customer set in context, hence we're able to access all the properties on the customer object and compare against it. I'll explain how the propertyChain is used in a later section.
The second rule checks to make sure the user isn't part of a specific Experiment. The way that's done is by ensuring the user's variantId for that experiment is either set to "A" or "EXCLUDED_FROM_EXPERIMENT" (in that case "B" variantId means that the user is participating in the experiment).
Last but not least is the check to make sure this isn't a new user and that they've requested from us before.

Data Models

RequirementRule.js: Is the entry point data for this Requirement system. In the examples above, the property called "requirement" is of this type.

'use strict'

const Archetype = require('archetype')
const assert = require('assert')
const RequirementProperty = require('./RequirementProperty')

const RequirementRule = new Archetype({
  property: {
    $type: RequirementProperty,
    $description: [
      'The property to evaluate or use'
    ]
  },
  comparisonValues: {
    $type: [RequirementProperty],
    $description: [
      'The value(s) to compare the users properties against to see if they belong in the requirement'
    ]
  },
  operand: {
    $type: 'string',
    $required: true,
    $enum: [
      'and', 'or', 'has', '!has', 'lessThan', 'lessThanOrEqual', 'greaterThan', 'greaterThanOrEqual',
      'lessThanAll', 'lessThanOrEqualAll', 'greaterThanAll', 'greaterThanOrEqualAll', 'versionLessThan',
      'versionLessThanOrEqual', 'versionGreaterThan', 'versionGreaterThanOrEqual'
    ],
    $description: [
      'The operand to check whether or not users fit into this audience based on the doc.property'
    ]
  },
  comment: {
    $type: 'string',
    $description: 'An internal comment to explain what the rule does.'
  }
}).compile('RequirementRule')

RequirementRule
  .path('nestedRules', {
    $type: [RequirementRule],
    $validate: (v, propTypes, doc) => assert.ok(['and', 'or'].includes(doc.operand), 'only and/or rules can contain nestedRules')
  }, { inPlace: true })
  .compile('SegmentationRule')

module.exports = RequirementRule

property: it's the type of property that this rule will contain. The type of that property is defined below. Please hold that thought, it'll make sense in a bit.

comparisonValues: as described in the comments above, these are the values that we'll be comparing our property with.
operand: the list of operations this system will support at the rule level.
comment: that's just to describe what the rule is about so we don't have to read the logic and make sense why we did it this way.
Something to note here is that this class can either just be a flat out property, or if it has an operand of type and or or then it needs to contain a nestedRules property as we have to have some rules defined in order to perform an and or or.

RequirementProperty.js: This is the class that describes what the requirement/rule's property entails. It can either be a query, a propertyChain or a constant.

'use strict'

const Archetype = require('archetype')
const PropertyOffset = require('./RequirementPropertyOffset')
const PropertyQuery = require('./RequirementPropertyQuery')
const { transformValue } = require('../../util/transformValues')

module.exports = new Archetype({
  query: {
    $type: PropertyQuery,
    $description: 'The query to execute to fetch that property.'
  },
  propertyChain: {
    $type: 'string',
    $description: [
      'The object path to use. e.g.: {{customer.numCompletedRequests}} or ',
      '{{result.variantId}}. Using result in this context uses the query result.'
    ]
  },
  constant: {
    $type: Archetype.Any, // This means it can be anything
    $transform: Archetype.matchType({ 'string': value => transformValue(value) }),
    $description: 'A constant to use. e.g.: 5 or "Karl"'
  },
  offset: {
    $type: PropertyOffset,
    $description: 'The offset to use when fetching the property.'
  }
}).compile('RequirementProperty')

query: is an object that has everything it needs in order perform a database query (PropertyQuery data model will follow).

propertyChain: This will always enclose a string using this annotation "{{...}}" and it can reference whatever instances we have in context. More on how to pass the context later. Imagine you know that you have access to the customer, so you can just do "{{customer.firstName}}" for example.
constant: Another way of describing this property by saying that we simply want it to be a constant that is some integer value e.g.: 5 or a string "Karl" or a date.
offset: This will allow us to offset one of the properties above by a specific amount. More on what the available options later. But the idea is that you can say "Take the number of completed requests the customer made, but offset them by 1". So for that example you'll end up with: propertyChain: "{{customer.numCompletedRequests}}", "offset": -1.

RequirementPropertyOffset.js: This is the offset property's type that can offset whatever the property is by a factor.

'use strict'

const Archetype = require('archetype')

module.exports = new Archetype({
  number: {
    $type: 'number',
    $description: 'The offset is defined as an actual number'
  },
  minutes: {
    $type: 'number',
    $description: 'The offset is a number that will use moment as offset.'
  },
  days: {
    $type: 'number',
    $description: 'The offset is a number that will use moment as offset.'
  }
}).compile('RequirementPropertyOffset')

This is pretty much self explanatory, the idea is that we can either set a value in minutes, days or a number. If you'd like to maybe extend this class by adding a custom string value to say: Offset the {{customer.firstName}} by some string, you can. It's odd but you can 🙂.

RequirementPropertyQuery.js: This is the query property type that allows us to specify a collection name, what query params to use, how to sort, etc.

'use strict'

const Archetype = require('archetype')
const QueryWhere = require('./RequirementPropertyQueryWhere')
const ValidCollections = require('../../util/validRequirementQueryCollections')
const ValidMethods = require('../../util/validRequirementQueryMethods')

module.exports = new Archetype({
  collection: {
    $type: 'string',
    $enum: ValidCollections,
    $description: 'The collection name to query.',
    $required: true
  },
  method: {
    $type: 'string',
    $enum: ValidMethods,
    $description: 'The method to use in the query.',
    $required: true
  },
  where: {
    $type: QueryWhere,
    $description: 'The where clause to use in the query.',
    $required: true
  },
  sort: {
    $type: Object,
    $description: [
      'The property to sort on after executing the query. This can also ',
      'be used when performing a "findOne" query, where it\'ll get the ',
      'first element in the resulting array.'
    ]
  }
}).compile('RequirementPropertyQuery')

collection: Is the list of valid collections we're exposing in this system. Perhaps we don't want to allow querying on some sensitive database collection.
method: is the query method to use find, findOne, count, etc.
where: object holding the query parameters (e.g.: where customerId is something and customer.firstName is something else).
sort: This is pretty much saying if we'd like to sort them, how we should sort them.

validRequirementQueryCollection.js: Used in the RequirementPropertyQuery above.

'use strict'

module.exports = [
  'Customer',
  'CustomerExperiment',
  ...
]

validRequirementQueryMethods.js: Used in the RequirementPropertyQuery above.

'use strict'

module.exports = [
  'findOne',
  'count'
]

RequirementPropertyQueryWhere.js: Used in the RequirementPropertyQuery above.

'use strict'

const Archetype = require('archetype')
const { transformObjectValues } = require('../../util/transformValues')

module.exports = new Archetype({
  and: {
    $type: Object,
    $transform: Archetype.matchType({ 'object': obj => transformObjectValues(obj) }),
    $description: 'The clause to use in the query.'
  },
  or: {
    $type: Object,
    $transform: Archetype.matchType({ 'object': obj => transformObjectValues(obj) }),
    $description: 'The "$or" clause to use in the query.'
  }
}).compile('RequirementPropertyQueryWhere')

This is where you can define that you have a query property with $or or $and as query parameters.

Building The Requirement System

Now we get to the more interesting bits of this system, where we write some logic that will consume these data models, and provide an entry point to use to start evaluating all those conditions.

Entry Point

Somewhere in your code, you'd want to evaluate those conditions and based on those, perform some action. Example:

// 1
...
const customer = {
  name: 'Karl',
  numCompletedRequests: 4
}

const promotion = {
  name: '$5 off next order',
  claimRequirement: {
    "comment": "Customer has ordered before",
    "operand": "greaterThan",
    "property": {
      "propertyChain": "{{customer.numCompletedRequests}}"
    },
    "comparisonValues": [{
      "constant": 0
    }]
  }
}

// 2
...
const didPass = evaluateRequirements({
  customer,
  promotion
})

if (didPass) {
  // perform something as we passed the requirements.
}

Let's pretend that we have a customer instance that we grabbed from the database somewhere in our business logic. And we also have a promotion that we fetched from the db. As you can see the promotion includes a claimRequirement property which defines our RequirementRule.
We pass the customer and promotion to our custom evaluateRequirements method defined below.

function evaluateRequirements (params) {
  // 1
  let {
    customer,
    promotion
  } = params

  // 2
  const supportedProps = generatedSupportedProperties(db)({
    customer,
    promotion
  })

  // 3
  return passesRequirement(db)(supportedProps, promotion.claimRequirement)
}

This is a custom method that every other system using our Requirement system needs to define in order to provide the "context" I've been referring to since the beginning. In this method you can do whatever you need in order populate all the supported properties you expect to use in your data model defined in the db.

We grab the customer/promotion we sent as parameters.
Generate supported properties will massage the customer's instance if needed, it might load a bunch of other collections from the db and link them to the customer, maybe check if the customer has an order and also include that. Pretty much whatever you want as the "context" needs to be generated in that method. Think about this method as a helper method that all the Requirement system uses will go through (e.g.: we expect that every customer in context will have the vehicles that are attached to that customer).
After massaging, populating and preparing our supported properties, let's call the meat of our system that will do a bunch of evaluations recursively in order to check whether we pass or fail the rules specified.

function generateSupportedProperties (params) {
  let {
    customer
  } = params

  const order = db.collection('Order').findOne({
    customerId: customer._id,
    status: 'ACTIVE'
  })

  const vehicle = db.collection('Vehicle').findOne({
    customerId: customer._id
  })

  if (vehicle) {
    customer.vehicle = vehicle
  }

  return {
    customer,
    order
  }
}

This is just a sample of what this method can look like, we just do some custom logic in order to either provide more properties we can support, link some properties to the customer (in that example we linked the vehicle) and return an order if we have one.

Core Logic

Here we will discuss what this passesRequirement() function is all about, how it loops through the rules, evaluate them, etc. Fasten your seatbelt it's gonna be a complex but fun ride🎢

function passesRequirement(db) {
  return function (supportedProps, requirement, loggingProps) {
    return co(function* () {
      // 1
      if (requirement.operand === 'and') {
        for (let rule of requirement.nestedRules) {
          // 2, if at least one of them doesn't pass
          const res = yield passesRequirement(db)(
            supportedProps,
            rule,
            loggingProps
          )
          if (!res) {
            return false
          }
        }
        return true
      } else if (requirement.operand === 'or') {  // 3
        for (let rule of requirement.nestedRules) {
          // 4, if at least one of them passes -> we're good
          const res = yield passesRequirement(db)(
            supportedProps,
            rule,
            loggingProps
          )
          if (res) {
            return true
          }
        }
        return false
      } else {
        // 5
        const evaluation = yield evaluateRule(db)(
          supportedProps,
          requirement,
          loggingProps
        )
        return evaluation
      }
    })
  }
}

That was the entry point that the users of the system will be calling. Based on the operand which is inside the rule, we’ll either ’and’ on the conditions or we ’or’ them:

We then loop through every nested rule within the nestedRules array and make sure all of them pass (in other words if at least one of them doesn’t pass, we failed the check).
Notice how we recursively call that method and evaluate everything within the rule to make sure it passes.
Otherwise if it’s an or operand, we still loop through the nestedRules array but make sure that at least one of the rules pass.
Same here, we recursively call the method to ensure everything within that rule pass.
Finally the exit condition of the recursion is to evaluate the rule (described below), which is the actual logic that compares our context data with some predefined conditions.

Let’s take a look at the evaluateRule() method:

const evaluateRule = (db) =>
  function (supportedProps, requirement, loggingProps) {
    return co(function* () {
      const { property, comparisonValues, operand } = requirement

      // replace that property by the result of a query, or direct replacement
      const { resultingValue, chainRemainder } = yield fetchProperty(db)(
        property,
        supportedProps,
        loggingProps
      )

      const resultingProperty = resultingValue

      // add the resulting property to the supportedProps so that we can re-use it
      // in the comparisonValues
      let supportedPropsWithResult = supportedProps
      if (resultingProperty) {
        supportedPropsWithResult.result = resultingProperty
      }

      // mutate the comparisonValues to dyamic variables if applicable.
      let mutatedComparisonValues = []
      for (let cv of comparisonValues) {
        const { resultingValue } = yield fetchProperty(db)(
          cv,
          supportedPropsWithResult,
          loggingProps
        )
        mutatedComparisonValues.push(resultingValue)
      }

      // If the resulting property is an array, let's loop
      // and recursively evaluate each of the items in the array
      // But first make sure we have a remainder of a chain
      // to keep traversing but on the items this time
      if (Array.isArray(resultingProperty) && chainRemainder) {
        for (let item of resultingProperty) {
          const itemEval = yield evaluateRule(db)(
            item,
            {
              operand,
              property: {
                propertyChain: `{{${chainRemainder}}}`,
              },
              // optimization: here the comparisonValues have
              // already been dynamically replaced, so just send
              // them to evaluate each item.
              comparisonValues: mutatedComparisonValues,
            },
            loggingProps
          )

          if (itemEval) {
            return itemEval
          }
        }
      }

      if (operand === 'has' || operand === '!has') {
        const has = mutatedComparisonValues.find((a) => a === resultingProperty)
        // note: Have to treat the "0" as a valid value
        return evaluateOperand(has != null, operand)
      } else if (
        [
          'lessThan',
          'lessThanOrEqual',
          'greaterThan',
          'greaterThanOrEqual',
          'versionLessThan',
          'versionLessThanOrEqual',
          'versionGreaterThan',
          'versionGreaterThanOrEqual',
          'contains',
        ].includes(operand)
      ) {
        for (let cv of mutatedComparisonValues) {
          // if at least one of them passes the evaluation
          // we're good
          if (evaluateOperand(resultingProperty, operand, cv)) {
            return true
          }
        }

        // nothing passed, let's fail it.
        return false
      } else if (
        [
          'lessThanAll',
          'lessThanOrEqualAll',
          'greaterThanAll',
          'greaterThanOrEqualAll',
          'containsAll',
        ].includes(operand)
      ) {
        // case of checking all of them
        const filteredValues = mutatedComparisonValues.filter(function (cv) {
          return evaluateOperand(resultingProperty, operand, cv)
        })

        // for now we have to have all the values in the comparisonValues
        // meeting the requirement.
        return filteredValues.length === mutatedComparisonValues.length
      } else {
        return false
      }
    })
  }

That method is very well documented, the gist of it is:

We first “fetch“ the property by calling the fetchProperty() method defined below. More on that to follow, but the idea is that we’ll end up with a resultingProperty value which could be a string, a number or a date.
We get the resultingProperty and chainRemainder from the return. The resultingProperty is well, the string, number or date explained above. The chainRemainder is there to let us know whether we were able to “consume” the dot “.” notation fully. Example: if we have our “propertyChain: {{customer.numCompletedRequestsss}}” and we end up with the resultingProperty = “customer” and chainRemainder = “numCompletedRequestsss”, we’ll know that because the chainRemainder hasn’t been consumed, and the resultingProperty is of type string it means that we must have a typo (in this case numCompletedRequestsss vs numCompletedRequests).
Next is to take that resultingProperty and insert it into our supportedPropsWithResult, which means that whatever context we sent initially at the time of calling the method, will now be accompanied with that result. This way we can use that result we just generated recursively in the neighboring rules.
Now we get to the comparisonValues, which are the values we will use to compare against our resultingValue mentioned above (the value got in the first bullet-point above). As you can see the comparisonValues are converted into a mutatedComparisonValues, that’s because every comparison value is made of our RequirementProperty. Which means that it can also be a query, constant or whatever our system supports. So we should “evaluate” the value first, then whatever the result is, we push it into our mutatedComparisonValues array. This is the array we’ll use to compare with our initial resultingProperty.
At this point we have the resultingProperty which could be let’s say 14 for the case of “customer.numCompletedRequests”. And we got the mutated comparison values to be [10, 14, 5] for example.
Next we check if our resultingProperty is of type array, and we got some chainRemainder left. Example if our propertyChain: “{{customer.friends.firstName}}”, we’ll detect that customer.friends is of type array, and our chainRemainder will be “firstName”, in that case we compare our resultingProperty to any of the friends’ first name and see if they match. Basically by recursively calling evaluateRule() but this time with our newly fetched properties.
If we don’t have a case of an array, we check whether the operand is ‘has’ or ‘!has’ (i.e.: whether they array contains at least one or none of them match), in other words at least one of the comparisonValues match my resultingProperty, or whether all of them don’t match. We use evaluateOperand() method defined below the fetchProperty(). I know this is getting out of hand 🤦‍♂️ — You need to take any of the examples at the top of this post and follow it alongside, it’ll make more sense.
If it’s not a case of a ‘has’ or ‘!has’, we group all the supported conditions (‘lessThan’, ’lessThanOrEqual’, etc.) in a way to perform the check easily on the array by saying “at least one of them is XYZ”.
Otherwise in the case of ‘lessThanAll’, ‘lessThanOrEqualAll’, basically anything with “all” in it, we group them into a check that makes sure that ”all of them pass“ in order to pass the overall check!

Now comes the fetchProperty() method:

const fetchProperty = (db) =>
  function (property, supportedProps, loggingProps) {
    return co(function* () {
      const { query, propertyChain, constant, offset } = property

      // if we got a constant, just use it as is
      // note: checking explicitely for null or undefined, as we don't want to
      // consider a 0 as null!
      if (constant != null) {
        return {
          resultingValue: handleOffset(
            // 06/24/2020: if the constant is stored as objectId, we usually want
            // to convert that into a string, as we can't compare objectIds together.
            // That was mostly added to cover for the recent transforms that take
            // the string and convert them to ObjectId before saving it in the db
            constant instanceof ObjectId ? constant.toString() : constant,
            offset
          ),
        }
      } else if (query) {
        // if we got a query to execute, let's do that first.
        const { collection, method, where, sort } = query

        // Set a limit on the find query
        const findLimit = 10

        // Allowing specific collections to be querried.
        if (!allowedQueryCollections.includes(collection)) {
          throw new BadRequest(
            `Querying "${collection}" is not allowed.
          \n\nOnly the following collections are allowed: ${allowedQueryCollections}`
          )
        }

        // Check for the methods (just to not fail silently)
        if (!allowedQueryMethods.includes(method)) {
          throw new BadRequest(
            `Query method "${collection}" is not allowed.
          \n\nOnly the following methods are allowed: ${allowedQueryMethods}`
          )
        }

        let conditions = {}
        if (where.and) {
          for (let prop in where.and) {
            // check if the prop has an elemMatch
            // to form the query.
            if (where.and[prop] && where.and[prop].elemMatch) {
              const { replacement } = replacementVariablesParser.parseContent(
                where.and[prop].elemMatch,
                supportedProps,
                loggingProps
              )

              conditions[prop] = {
                $elemMatch: replacement,
              }
            } else {
              const { replacement } = replacementVariablesParser.parseContent(
                where.and[prop],
                supportedProps,
                loggingProps
              )

              conditions[prop] = replacement
            }
          }
        } else if (where.or) {
          let orConditions = []

          for (let prop in where.or) {
            const condition = {}

            // Check if there is an elemMatch
            // for the query
            if (where.or[prop] && where.and[prop].elemMatch) {
              const { replacement } = replacementVariablesParser.parseContent(
                where.or[prop].elemMatch,
                supportedProps,
                loggingProps
              )

              condition[prop] = {
                $elemMatch: replacement,
              }
            } else {
              const { replacement } = replacementVariablesParser.parseContent(
                where.or[prop],
                supportedProps,
                loggingProps
              )

              condition[prop] = replacement
            }

            orConditions.push(condition)
          }

          conditions = {
            $or: orConditions,
          }
        }

        let result, resultChainRemainder

        // if we have a findOne + sort, always take the first element for now.
        if (method === 'findOne' && sort) {
          result = yield db
            .collection(collection)
            .find(conditions)
            .sort(sort)
            .limit(findLimit)
          // grab the first element if it exists
          if (result && result.length > 0) {
            result = result[0]
          } else {
            // we wanted only 1 from that array but got nothing back,
            // let's clear the result so we don't try to dynamically parse it.
            result = null
          }
        } else if (method === 'findOne') {
          result = yield db.collection(collection).findOne(conditions)
        } else if (method === 'count') {
          result = yield db.collection(collection).count(conditions)
        }
        // WARNING: Don't allow using find to not choke the db.
        // else if (method === 'find') {
        //   result = sort ? await db.collection(collection).find(conditions).sort(sort)
        //     : await db.collection(collection).find(conditions)
        // }

        // note: We want to include a result = "0" here.
        if (result != null && propertyChain) {
          // then parse it using the propertyChain
          const { replacement, chainRemainder } =
            replacementVariablesParser.parseContent(
              propertyChain,
              { result },
              loggingProps
            )

          result = replacement
          resultChainRemainder = chainRemainder
        }

        return {
          resultingValue: handleOffset(result, offset),
          chainRemainder: resultChainRemainder,
        }
      } else if (propertyChain) {
        // we got no query, let's leave.
        const { replacement, chainRemainder } =
          replacementVariablesParser.parseContent(
            propertyChain,
            supportedProps,
            loggingProps
          )

        return {
          resultingValue: handleOffset(replacement, offset),
          chainRemainder,
        }
      }

      // default catch to return the value as is
      return {
        resultingValue: property,
      }
    })
  }

That method is well documented as well, the long story short is:

Check if it’s a constant value we have, then just offset it by some constant value if needed. Example adding a few seconds to some date, or a few minutes, days, etc. Or just subtract a number from the value.
Otherwise if it’s a query, then take all the properties that need to be associated with a query in order to perform an actual database query.
- As you’ll notice that we have a limit set to 10 just so we don’t allow queries that could take our database down.
- You’ll also notice that there are some allowedQueryCollections so we can have some collections to be off-limit (sensitive info or whatever they may be)
- As well as allowedQueryMethods so we can’t do a delete or deleteMany on the database.
- Then while we’re on that same query evaluation, we check if it’s an and or an or to construct the query. Notice how that will call replacementVariablesParser.parseContent(). This is method that will convert the “{{customer.numCompletedRequests}}” to a value.
- We perform the database query.
- Then attempt to go deeper in the propertyChain, and grab whatever is left in the result and resultChainRemainder. The chain remainder is whatever was left from the dot notation of the `propertyChain`.
- Finally we apply any offset if it exists to the result (example: taking the customer.numCompletedRequests - 3.)
Otherwise if it’s a propertyChain (example: “{{customer.numCompletedRequests}}”)
- Just perform the parseContent() to convert that string above into its corresponding value.
- Apply offset as well.

Following are the more “helper” methods used in the methods defined above.

evaluateOperand() helper method:

function evaluateOperand(value, operand, compareTo) {
  if (operand === 'has') {
    return !!value
  } else if (operand === '!has') {
    return !value
  } else if (operand === 'greaterThan' || operand === 'greaterThanAll') {
    return value != null && value > compareTo
  } else if (
    operand === 'greaterThanOrEqual' ||
    operand === 'greaterThanOrEqualAll'
  ) {
    return value != null && value >= compareTo
  } else if (operand === 'lessThan' || operand === 'lessThanAll') {
    return value != null && value < compareTo
  } else if (
    operand === 'lessThanOrEqual' ||
    operand === 'lessThanOrEqualAll'
  ) {
    return value != null && value <= compareTo
  } else if (operand === 'versionGreaterThan') {
    return value != null && semver.gt(value, compareTo)
  } else if (operand === 'versionGreaterThanOrEqual') {
    return value != null && semver.gte(value, compareTo)
  } else if (operand === 'versionLessThan') {
    return value != null && semver.lt(value, compareTo)
  } else if (operand === 'versionLessThanOrEqual') {
    return value != null && semver.lte(value, compareTo)
  } else if (operand === 'contains' || operand === 'containsAll') {
    return Array.isArray(value) && value.includes(compareTo)
  } else {
    throw new BadRequest(`Operand "${operand}" is not supported.`)
  }
}

That’s the method that performs the actual comparison between the mutatedComparisonValues and the resultingProperty we previously mentioned. Not much to elaborate there, as it’s just a conversion between the enum type i.e.: ‘lessThan’ and performing the actual math ‘<‘ between left and right hand side operands.

handleOffset() helper method:

function handleOffset(value, offset) {
  let result = value
  if (offset) {
    // check if the value is of type date, to use time offset
    if (offset.minutes) {
      return moment(value).add(offset.minutes, 'minutes').toDate()
    } else if (offset.days) {
      return moment(value).add(offset.days, 'days').toDate()
    } else if (offset.number) {
      return value + offset.number
    }
  }
  return result
}

This method will take a value and apply an offset to it. If the offset has minutes, we use moment library in order to add some minutes to our value which in this case should be a date. Otherwise we do days, otherwise we apply an offset to a number. If you’d like to get fancier and support more offsets, you can!

replacementVariablesParser.parseContent() helper method:

'use strict'

const logger = require('../../util/logging')

exports.parseContent = (content, supportedProps, loggingProps) => {
  let updatedContent = content
  let foundAtLeastOne = false
  let foundError = false
  let foundChainRemainder

  const regex = RegExp('{{(.*?)}}', 'g')
  let match
  while ((match = regex.exec(content)) !== null) {
    // The part of the match that corresponds to the parentheses
    if (match[1]) {
      // split using the '.'
      const split = match[1].split('.')

      const { replacement, error, chainRemainder } =
        exports.getReplacementForSupportedVariables(
          split,
          supportedProps,
          loggingProps
        )

      // if we found at least 1 error, let's keep track
      // for later
      if (error) {
        foundError = error
        foundChainRemainder = chainRemainder
      }

      // if we had a successful replacement, let's replace
      // the whole match (including the parentheses) with the replacement
      if (replacement !== null && replacement !== undefined) {
        // if the replacement yielded to a string, let's swap
        // the updatedcontent's match with the result
        if (typeof replacement === 'string') {
          updatedContent = updatedContent.replace(match[0], replacement)
        } else {
          // otherwise yielded to a non string, let's just swap
          // the updatedcontent altogether with the result
          updatedContent = replacement
        }

        // tag that we found at least one for the recursion
        foundAtLeastOne = true
      }
    }
  }

  // If we couldn't perform any replacement this time, let's stop recursing.
  if (!foundAtLeastOne) {
    return {
      replacement: updatedContent,
      chainRemainder: foundChainRemainder,
      error: foundError,
    }
  } else {
    // otherwise try to dynamically replace again...
    const { replacement, error } = exports.parseContent(
      updatedContent,
      supportedProps,
      loggingProps
    )

    return {
      replacement,
      chainRemainder: foundChainRemainder,
      // Take either the current error, or the new one. (We don't want to
      // allow the lower recursions to override a positive error)
      error: foundError || error,
    }
  }
}

This method above is pretty well documented. The idea is that it’ll look for the dot notation and attempt to convert that chain “{{customer.numCompletedRequests}}” using the supportedProps provided (which is where we send the customer js object which has the numCompletedRequests property in it). That will yield to the deepest chain we can form before hitting an undefined and not being able to use the dot notation to get any deeper. And return the chainRemainder (whatever is left) back.

replacementVariablesParser.getReplacementForSupportedVariables() helper method:

exports.getReplacementForSupportedVariables = (
  chain,
  supportedProps,
  loggingProps = {}
) => {
  // make sure we support that main variable (customer, serviceLocation, etc...)
  let replacement = supportedProps[chain[0]]
  // keep track of whether there was an error
  let error = false
  // keep track of whether there was a remainder
  // of the chain we couldn't find
  let chainRemainder

  // now loop through the chain to drill deeper
  if (replacement != null) {
    if (chain.length > 1) {
      let current
      // start from the second link in the chain and forward, as we
      // already started with the base variable
      for (let i = 1; i < chain.length; i++) {
        // move forward in the chain
        current = replacement[chain[i]]

        // check if it's not supported, to abort
        // note: don't use !current, as we want to consider a "0" as something valid.
        if (current === null || current === undefined) {
          // but first check to see this is a method or something we can handle
          const mutatedReplacement = checkForMethodCalls(replacement, chain[i])
          if (mutatedReplacement) {
            // we got something (function call or something)
            current = mutatedReplacement
          } else {
            // combine everything that is left from here forward
            chainRemainder = chain.splice(i, chain.length - i).join('.')
            // we couldn't find a match, let's log it.
            logger.error(
              'Error when parsing dynamic content chain link.',
              Object.assign(loggingProps, {
                chainLink: chainRemainder,
                property: chain[0],
              })
            )
            error = true
            break
          }
        }

        // grab whatever is in the current one.
        replacement = current
      }
    }
  } else {
    // we couldn't find a match, let's log it.
    logger.error(
      'Error when parsing dynamic content.',
      Object.assign(loggingProps, {
        content: chain.join('.'),
        supportedProps: JSON.stringify(supportedProps),
      })
    )
    error = true
  }

  return { replacement, error, chainRemainder }
}

function checkForMethodCalls(property, chainLink) {
  let mutated

  if (chainLink === 'toString()') {
    mutated = property.toString()
  }

  return mutated
}

Same goes here, the comments should cover it. That method is used in the previous method above it. A small nuance here is that we call the checkForMethodCalls just to allow us to do things like “{{customer._id.toString()}}” — This way we can call a method on the result by checking if the chain link we’re evaluating now is a method we support, then we actually call that method on the object.

This concludes our system from data models to implementation. I know it’s been a wild ride and most of it seems a bit too complex, but believe me once you incorporate it slowly and you have an example alongside, it’ll all make sense. I mean at the end of the day you’re building a very flexible system that can handle pretty complex rules and requirements that is all driven by the data stored in mongodb, without having to rewrite a single line of code! That’s pretty neat if you ask me 😉

I do hope that post provided some value to you, even if you don’t end up incorporating it, seeing all these recursions and how you can have data driven business logic is an interesting topic to be exposed to as an engineer! Cheers 🍻

nodesystem-architecture