How to Modify All Values of an Object in DataWeave

An Overview

By: Joshua Erney

Recently, I was getting data from a stored procedure that contained a very large amount of whitespace on certain fields but not on others, and, occasionally, those fields were null. I wanted to remove this white space on these fields. Below, I will document the process to get to the final iteration of the code by using more and more functional programming concepts as I go. Here was my first go at it:

First Iteration:

%dw 1.0
%output application/java
 
%function removeWhitespace (v) trim v when v is :string otherwise v
---
payload map {
  field1: removeWhitespace($.field1),
  field2: removeWhitespace($.field2)
...
}

This works. But it has two problems: it obscures the mapping by adding a repetitive function call for EVERY field and, if you don’t want to do every field, you would need to individually identify which fields have extra whitespace and which ones don’t. This is time consuming and potentially impossible given that this could change from row-to-row in the database. So, this solution might work now, but it might not work for every field. Additionally, if things change, it is incredibly brittle. Here’s my second iteration:

Second Iteration

%dw 1.0
%output application/java
 
%function removeWhitespaceFromValues(obj)
  obj mapObject {($$): trim $ when $ is :string otherwise $}
 
%var object = removeWhitespaceFromValues (payload)
---
object map {
  field1: $.field1,
  field2: $.field2
...
}

Awesome! We no longer have to individually identify which fields have a bunch of white space because the function doesn’t care. It will trim every value that is a string, which is exactly what we want. But… could it be better? What if someone wanted to use this code in the future to do the same thing to a JSON object with nested objects and lists? The second iteration of the function will not accomplish this; it will not apply the trim to nested objects and arrays. This next sample uses the match operator, which brings pattern matching to DataWeave. If this is your first exposure to either match or pattern matching, you can find out more about the match operator using these MuleSoft documents , and more about pattern matching in general here. Now let’s take a look at the code.

Third Iteration

%dw 1.0
%output application/java
 
%function removeWhitespace (e)
  e match {
    :array  -> $ map removeWhiteSpaceFromValues($),
    :object -> $ mapObject {($$): removeWhiteSpaceFromValues($)},
    default -> trim $ when $ is :string otherwise $
    }
%var object = removeWhitespaceFromValues( payload )
---
...


Cool. Now we have a function that will remove the white space from every value in an element that is a string, including deeply-nested elements. You might think we’re done. But we can do a lot better using something called higher-order functions. In other words, we’re going to pass a function into our existing function to specify exactly how we want it to work. Check it out (thanks to Leandro Shokida at MuleSoft for his help with this function):

Final Iteration

%dw 1.0
%output application/java
 
%function applyToValues(e, fn)
  e match {
    :array  -> $ map applyToValues($, fn),
    :object -> $ mapObject {($$): applyToValues($, fn)},
    default -> fn($)
}
 
%function trimWhitespace(v)
  trim v when v is :string otherwise v
 
%var object = applyToValues(payload, trimWhitespace)
---
...

This effectively makes the act of looping through every value in an element completely generic “applyToValues“. From here, we can define exactly what we want to happen for each value in the element “trimWhitespace“. We’ve effectively separated the concern of targeting every value in an element, with the concern of what to do with that value. What if we wanted to do something other than trim each value in the object? Just change the function you pass in. Maybe you want to trim the value if it’s a string, and increment it if it’s a number. Let’s see what that would look like:

Third Iteration (New Functionality)

%dw 1.0
%output application/java
 
%function applyToValues(e, fn)
  e match {
    :array  -> $ map applyToValues($, fn),
    :object -> $ mapObject {($$): applyToValues($, fn)},
    default -> fn($)
}
 
%function trimOrIncrement(v)
  v match {
    :string -> trim v,
    :number -> v + 1,
    default -> v
}
 
%var object = applyToValues(payload, trimOrIncrement)
---
...

Notice the most important thing here, the applyToValues function did not need to change at all. The only thing we changed was the function we passed into it. One last point, we don’t even need to give our function a name; we can create the second argument to applyToValues on the spot using a lambda or anonymous function. Here we will use a lambda to increment the value if it’s a number:

Third Iteration (lambda)

%dw 1.0
%output application/java
 
%function applyToValues(e, fn)
  e match {
    :array  -> $ map applyToValues($, fn),
    :object -> $ mapObject {($$): applyToValues($, fn)},
    default -> fn($)
}
 
%var object = applyToValues(payload, ((v) ->
  v + 1 when v is :number otherwise v))
---
...

In Summary

In conclusion, the idea is that DataWeave is flexible enough that you can have a chunk of code that steps through all the values of an element, and that code can be reused across projects (see Mulesoft documentation on readUrl). At the same time, you don’t need to permanently wire-in how each value is modified (i.e. it could be a trim, it could remove all spaces, it could add 1 to every number, etc). This kind of functionality is easily available to you in DataWeave, take advantage!

Josh Erney  has been working as a software engineer at Mountain State Software Solutions (ArganoMS3) since November 2016, specializing in APIs, software integration, and MuleSoft products.