distinctBy() in DataWeave 2.0 — Remove Duplicate Records Easily

Duplicate records are common in real-world MuleSoft integrations. An API may return the same customer multiple times, a database query may contain duplicates, or an upstream system may send repeated messages.

DataWeave provides the distinctBy() function to remove duplicates based on a field or expression.

---

🔹 What Is distinctBy()?

The distinctBy() function returns an array containing distinct elements based on the value returned by the expression provided.

The basic syntax is:

array distinctBy expression

For example:

payload distinctBy $.email

This keeps one record for each unique email address.

---

📥 Input JSON

[
  {
    "id": 101,
    "name": "John Doe",
    "email": "john@example.com",
    "department": "IT"
  },
  {
    "id": 102,
    "name": "Alex",
    "email": "alex@example.com",
    "department": "HR"
  },
  {
    "id": 103,
    "name": "John Doe",
    "email": "john@example.com",
    "department": "IT"
  },
  {
    "id": 104,
    "name": "Sam",
    "email": "sam@example.com",
    "department": "Finance"
  }
]

Notice that john@example.com appears twice.

---

💻 DataWeave Solution

%dw 2.0
output application/json
---
payload distinctBy $.email

---

📤 Output

[
  {
    "id": 101,
    "name": "John Doe",
    "email": "john@example.com",
    "department": "IT"
  },
  {
    "id": 102,
    "name": "Alex",
    "email": "alex@example.com",
    "department": "HR"
  },
  {
    "id": 104,
    "name": "Sam",
    "email": "sam@example.com",
    "department": "Finance"
  }
]

The duplicate record for john@example.com has been removed.

---

🔍 How Does It Work?

Step 1 — Start with the Array

DataWeave receives an array containing customer or employee records.

Step 2 — Select the Unique Field

distinctBy $.email

The expression $.email tells DataWeave which value should be used to determine uniqueness.

Step 3 — Remove Duplicates

Records with the same email value are treated as duplicates and only one record is retained.

---

🔹 distinctBy() Using Multiple Fields

Sometimes uniqueness depends on more than one field.

For example, suppose an order should be considered unique based on both customer ID and product ID.

%dw 2.0
output application/json
---
payload distinctBy {
  customerId: $.customerId,
  productId: $.productId
}

This allows you to define uniqueness using a combination of values.

---

🏢 Real-World MuleSoft Example

Imagine a MuleSoft application receives customer records from multiple source systems:

• Salesforce
• CRM database
• Legacy application
• External customer API

The combined response may contain duplicate customers. If email is the business key, we can remove duplicates before sending the records to the downstream system.

%dw 2.0
output application/json
---
payload distinctBy $.email

This is a simple but useful pattern in enterprise data integration.

---

🔹 distinctBy() vs filter()

Function	Purpose
distinctBy()	Removes duplicate elements based on an expression.
filter()	Keeps elements that satisfy a condition.

Think of it this way:

filter()     → "Does this record satisfy my condition?"

distinctBy() → "Have I already seen this unique value?"

---

🔹 distinctBy() vs groupBy()

These functions are also commonly confused.

• distinctBy() → removes duplicate records.
• groupBy() → organizes records into groups.

Example:

payload distinctBy $.department

keeps one record for each distinct department value.

payload groupBy $.department

creates groups containing the records belonging to each department.

---

🎯 MuleSoft Interview Question

How do you remove duplicate records from an array in DataWeave?

Answer: Use the distinctBy() function and provide the field or expression that determines uniqueness.

%dw 2.0
output application/json
---
payload distinctBy $.email

---

⚠️ Common Mistake

A common mistake is using distinctBy() without thinking about what actually defines a unique business record.

For example, two records may have different IDs but represent the same customer because they share the same business key such as email or customer number.

Always identify the correct business key before removing duplicates.

---

💡 Production Tip

For very large datasets, consider the memory implications of performing deduplication in memory. Depending on the integration design, database queries, batch processing, pagination or upstream deduplication may be more appropriate.

---

🚀 Final Takeaway

distinctBy() = Remove duplicates based on a unique expression.

The most common pattern is:

payload distinctBy $.businessKey

Mastering distinctBy() is especially useful when working with customer synchronization, order processing, API aggregation and database-to-API integrations.

---

📚 Want to Practice More DataWeave?

Practice DataWeave with 1000 real-world problems with complete solutions, from beginner concepts to advanced enterprise transformations.

👉 Get 1000 Problems to DataWeave Mastery →

---

Follow Digital Tech eBooks for more MuleSoft, DataWeave, API-led connectivity, connectors, deployment and real-world integration tutorials.