Batch enrichment guide

ⓘ Reading Time: 5 mins

Batch enrichment allows you to process large volumes of transactions efficiently. Instead of making individual API calls for each transaction, you can submit up to 50,000 transactions in a single request and retrieve the enriched results once processing is complete. We have two endpoints for batch card enrichment:

/batches/transactions/cards/enrich
/batches/transactions/cards/enrich/parse

These batch endpoints use the same enrichment logic as their single-transaction counterparts. The /batches/transactions/cards/enrich endpoint applies the same enrichment as /transactions/cards/enrich, while /batches/transactions/cards/enrich/parse performs the same parsing and enrichment as /transactions/cards/enrich/parse.

Getting Started

Let's walk through the process of enriching a batch of transactions:

1. First, submit your batch of transactions:

import requests
import time

# Prepare your batch of transactions
transactions = [
    {
        "userId": "user_123",
        "merchantName": "Walmart",
        "amount": 50.23,
        "currencyCode": "USD",
        "occurredAt": "2024-03-21T15:30:00Z",
        "categoryType": "MCC",
        "categoryCode": "5411",
        "transactionId": "tx_123",  # Must be unique within the batch
        "location": {
            "city": "Seattle",
            "region": "WA",
            "country": "USA"
        }
    },
    # ... more transactions ...
]

# Submit the batch (without a callbackUrl)
response = requests.post(
    "https://east.sandbox.spade.com/batches/transactions/cards/enrich",
    json={"transactions": transactions},
    headers={"X-Api-Key": "<Your API Key Here>"}
)

# Or submit with a callbackUrl
response = requests.post(
    "https://east.sandbox.spade.com/batches/transactions/cards/enrich",
    json={
        "transactions": transactions,
        "callbackUrl": "https://your-callback-endpoint.com/batch-notifications"
    },
    headers={"X-Api-Key": "<Your API Key Here>"}
)

batch_id = response.json()["batchId"]
print(f"Batch submitted with ID: {batch_id}")

Notice that we hold on to the batchId returned from the submission request.
This ID is required to check the status and retrieve the results of the batch enrichment job.

2. Receive notification of completion or check status:

You have two options to determine when your batch is complete.

Option A: Receive a callback notification (recommended)

If you provided a callbackUrl in your batch submission, we'll send a POST request to that URL when processing completes.

The callback will include a JSON payload with:

batchId: The ID of the completed batch
status: Either "completed" or "failed"

We will also include a token in the X-Webhook-Token header that you should compare against the callback token provided to you by your Spade representative. Contact your Spade representative if you do not have this token.

Here is an example Flask endpoint you could use to receive callbacks.

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/batch-notifications', methods=['POST'])
def batch_notification():
    webhook_token = request.headers.get('X-Webhook-Token')
    expected_token = "your-callback-token-from-spade"  # Store this securely

    if not webhook_token or webhook_token != expected_token:
        raise InvalidTokenError()

    data = request.json
    batch_id = data["batchId"]
    status = data["status"]


    if status == "completed":
        fetch_batch_results(batch_id)
    elif status == "failed":
        log_batch_failure(batch_id)

Option B: Poll for status

If you don't use callbacks, you can poll the status endpoint.

def wait_for_batch_completion(batch_id: str, delay_seconds: int = 60):
    """Wait for a batch job to complete, checking status periodically"""

    status = "pending"

    while True:
        response = requests.get(
            f"https://east.sandbox.spade.com/batches/{batch_id}",
            headers={"X-Api-Key": "<Your API Key Here>"}
        )
        status = response.json()["status"]

        print(f"Batch status: {status}")

        if status == "completed":
            return True
        elif status == "failed":
            raise Exception("Batch processing failed")

        time.sleep(delay_seconds)

    raise Exception("Timeout waiting for batch completion")

wait_for_batch_completion(batch_id)

3. Once the batch is complete, retrieve the results using the `/results` endpoint

response = requests.get(
    f"https://east.sandbox.spade.com/batches/{batch_id}/results",
    headers={"X-Api-Key": "<Your API Key Here>"}
)

results = response.json()["results"]

for enriched_transaction in results:
    print(f"Enriched transaction: {enriched_transaction['enrichmentId']}")
    # ... process the enrichment ...

Understanding Batch Status

A batch job can be in one of four states:

pending: The batch has been accepted but processing hasn't started
running: The batch is currently being processed
completed: All transactions have been enriched and results are ready
failed: The batch encountered an error and couldn't be processed

⚠
The /batches/{batchId}/results endpoint will only return results when the batch status is completed. If you request results before completion, you'll receive a 202 status code indicating that the results aren't yet available.

Best Practices

Unique Transaction IDs: Ensure each transaction in your batch has a unique transactionId. Results are not guaranteed to be returned in the order in which they were submitted, so you can use the transactionId to correlate transactions in the results with the original requests. Duplicate IDs will cause the batch enrichment request to fail.
Callbacks vs. Polling: Use callbacks where possible instead of polling. This reduces unnecessary API calls and lets your system respond to completion events more efficiently.
Status Polling: If you use the polling strategy, implement exponential backoff to avoid unnecessary API calls:

def wait_for_batch_completion_with_backoff(batch_id, max_attempts=10, initial_delay=60):
    """Wait for batch completion with exponential backoff"""
    delay = initial_delay

    while True:
        response = requests.get(
            f"https://east.sandbox.spade.com/batches/{batch_id}",
            headers={"X-Api-Key": "<Your API Key Here>"}
        )
        status = response.json()["status"]

        if status == "completed":
            return True
        elif status == "failed":
            raise Exception("Batch processing failed")

        print(f"Sleeping for {delay} seconds before checking status again")
        time.sleep(delay)
        delay = max(delay * 2, 60)

    raise Exception("Timeout waiting for batch completion")

Timeouts: Uploading large batches can take a while, so we recommend increasing the timeout of your client. We support up to a 120 second timeout window.
Error Handling: There are two classes of errors you will want to handle. The first is the errors you may encounter when submitting the batch via the /batches/transactions/cards/enrich endpoint. The second class of errors can be returned as individual payloads in the results array of the /batches/{batchId}/results endpoint. For example, you may see a 400 in the statusCode for a single request in the batch if the request body was missing a required field, while other requests in the batch return a 200 statusCode. In the event that a request resulted in an error, the correspond entry in the results array will contain an errors object. The errors object contains an array of objects explaining what caused the error to occur. Here's an example of what the errors object could look like:

errors:
    {
        'categoryCode': ['This field is required.']
    }

Note that even in the event of an error, the result will still contain the transactionId key within the transactionInfo object and the customAttributes object. You can use the data present in these objects to determine which payload within your input batch resulted in an error. See our enrichment guide for more details on handling errors from our API.

Using Batch Enrichment with DE43 Data

If you have unparsed DE43 data, you can use the /batches/transactions/cards/enrich/parse endpoint instead. This endpoint works exactly the same way, but allows you to omit some fields which are required in the /batches/transactions/cards/enrich endpoint and include DE43 data that will be parsed before enrichment. Under the hood, this endpoint behaves like the /transactions/cards/enrich/parse endpoint.

transactions = [
    {
        "userId": "user_123",
        "de43": "WALMART              SEATTLE     WAUS",
        "amount": 50.23,
        "currencyCode": "USD",
        "occurredAt": "2024-03-21T15:30:00Z",
        "categoryType": "MCC",
        "categoryCode": "5411",
        "transactionId": "tx_123",
    }
]

response = requests.post(
    "https://east.sandbox.spade.com/batches/transactions/cards/enrich/parse",
    json={"transactions": transactions},
    headers={"X-Api-Key": "<Your API Key Here>"}
)

Additional Resources

Getting Started Guide

Getting Started

1. First, submit your batch of transactions:

2. Receive notification of completion or check status:

3. Once the batch is complete, retrieve the results using the /results endpoint

Understanding Batch Status

⚠The /batches/{batchId}/results endpoint will only return results when the batch status is completed. If you request results before completion, you'll receive a 202 status code indicating that the results aren't yet available.

Best Practices

Using Batch Enrichment with DE43 Data

Additional Resources

3. Once the batch is complete, retrieve the results using the `/results` endpoint

⚠
The `/batches/{batchId}/results` endpoint will only return results when the batch status is `completed`. If you request results before completion, you'll receive a 202 status code indicating that the results aren't yet available.