How to test if your error tracker is leaking PII (run this on your last 1,000 events)

You're scanning a fresh production error in your tracker when you see it: a customer's full email address and hashed password sitting in the request.body of a breadcrumb. It's not just a bug; it's a compliance violation that is already synced to a third-party cloud. The "default" redaction settings failed, and now you have to figure out how many other thousands of records are sitting there like a ticking liability.

Most engineering teams treat PII (Personally Identifiable Information) scrubbing as a "set it and forget it" configuration. You check the sendDefaultPii: false box in your SDK init, maybe add a few common keys like password or token to a blacklist, and assume the problem is solved. But software evolves. Your schemas change, new third-party integrations are added, and developers inadvertently log "extra" context to help debug tricky issues. Every one of these changes is a potential leak vector.

If you haven't audited your error logs in the last six months, you are almost certainly leaking sensitive data. This isn't just about being "clean"; it's about GDPR Article 32 and OWASP standards. This post provides a repeatable, technical audit recipe—extracting a 1,000-event sample and running a gap analysis—to prove your redaction rules actually work before a compliance auditor asks for the same report.

The "Default Scrubber" Delusion

Why do standard SDK settings fail? The most common reason is that "default" scrubbing is typically designed to catch the obvious: headers like Authorization, cookies, and maybe some top-level fields in a standard web request. However, modern applications are messy. Data doesn't just live in the Authorization header; it lives in the breadcrumbs, the metadata, and the custom tags your team adds to "make debugging easier."

Why breadcrumbs are the primary leak vector

Breadcrumbs are a timeline of events leading up to a crash. They often include UI interactions (clicks, input changes) and network requests. While your SDK might scrub the body of the final error-causing request, it often leaves the body or url parameters of the *preceding* 10 successful requests untouched. If a user successfully logged in three steps before an unrelated crash, their credentials might be sitting in a fetch breadcrumb, completely bypassing your main error filter.

The "Context Creep" problem

Developers love context. We use setContext or setExtra to attach the current user's state to every error. Over time, that "state" object grows. What started as a simple user_id evolves into a full user_profile object containing names, phone numbers, and physical addresses. Because these are custom keys, the "default" SDK scrubber has no idea they should be treated as sensitive unless you explicitly tell it so.

Consider this common Sentry-style JSON payload. Even with sendDefaultPii: false, look at what survives in the extra block:

{
  "event_id": "84829fb...",
  "message": "TypeError: Cannot read property 'map' of undefined",
  "request": {
    "url": "https://api.example.com/v1/orders",
    "headers": {
      "Authorization": "[filtered]"
    }
  },
  "extra": {
    "last_saved_state": {
      "user_email": "jane.doe@personal-mail.com",
      "shipping_address": "123 Maple St, Springfield",
      "temp_auth_token": "sg_39201hf8..."
    }
  }
}

The Authorization header was caught, but the extra.last_saved_state was ignored because the SDK didn't recognize the keys. This is "Context Creep," and it's the #1 reason for PII leaks in production.

Phase 1: The 1,000-Event Extraction

To perform an audit, you need raw data. You cannot rely on the UI of your error tracker; most tools truncate long strings or hide certain metadata fields in the web view to keep things snappy. You need the full JSON payload for a statistically significant sample—at least 1,000 events.

Exporting raw JSON from your current provider

While some providers offer a "CSV Export," that is useless for PII auditing because it flattens the nested structures where PII hides. You need the raw JSON. The best way to get this is via the API. If you're using Sentry, you can use their /events/ endpoint to fetch the most recent events across all issues.

Using the API to bypass UI truncation

Don't try to copy-paste from the browser. Use a script to pull the data. This ensures you're getting the exact same bytes that your SDK sent to the server. Here is a simple curl loop to pull the last 1,000 events for a specific project. You'll need an internal integration token with event:read permissions.

#!/bin/bash
# extraction.sh
ORG="your-org-slug"
PROJECT="your-project-slug"
TOKEN="your-sentry-auth-token"
OUT_FILE="audit_dump.jsonl"

echo "Extracting last 1000 events..."

# Note: Sentry returns 100 events per page max
for i in {0..9}
do
  CURSOR=$( [ $i -eq 0 ] && echo "" || echo "&cursor=${NEXT_CURSOR}" )
  RESPONSE=$(curl -s -H "Authorization: Bearer $TOKEN" \
    "https://sentry.io/api/0/projects/$ORG/$PROJECT/events/?limit=100$CURSOR")
  
  # Append to our JSONL file (one event per line for easier grepping)
  echo "$RESPONSE" | jq -c '.[]' >> $OUT_FILE
  
  # Simple pagination logic would go here in a real script
  echo "Fetched $(( (i+1) * 100 )) events..."
done

echo "Extraction complete. Data saved to $OUT_FILE"

This script uses jq -c to transform the JSON array into a "JSON Lines" format. This is crucial because it allows us to use line-based shell tools like grep or awk to scan for patterns without loading the entire 100MB+ file into memory.

Phase 2: The "Grepping for Trouble" Audit

Now that you have your audit_dump.jsonl, it's time to find the leaks. We want to search for patterns that *should* have been redacted but weren't. We aren't looking for specific users; we're looking for the *presence* of PII formats.

The Regex Hitlist

Run these commands against your dump. If any of these return results, you have a leak. We'll use grep -E (Extended Regex) to find common PII formats.

1. Searching for Emails:
This is the most common leak. Emails often appear in URL parameters or breadcrumbs.

grep -E -o "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" audit_dump.jsonl | sort | uniq -c | sort -nr

2. Searching for JWTs:
JWTs are dangerous because they often contain user claims. They usually start with eyJ (the base64 for {"alg").

grep -E "eyJ[A-Za-z0-9-_=]+\.[A-Za-z0-9-_=]+\.?[A-Za-z0-9-_.+/=]*" audit_dump.jsonl

3. Searching for Credit Cards (Luhn-ish):
This will have false positives, but it's better to check. We look for 13-16 digit strings.

grep -E "\b[3456][0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}\b" audit_dump.jsonl

Identifying "Shadow PII"

Shadow PII refers to internal identifiers that aren't obviously "personal" but can be used to link an event to a real identity. Examples include IP addresses (which are PII under GDPR), session IDs, or even specific metadata like a device_name that contains a user's real name (e.g., "John Doe's iPhone"). Check for these by searching for keys that look like they might hold such data:

grep -E "(user_id|session_id|ip_address|customer_name|phone)" audit_dump.jsonl | head -n 20

If you see raw values instead of [filtered] or ********, your configuration is missing these keys.

Phase 3: The Gap Analysis

Once you've found a few leaks, you need to understand *why* they happened. This is where you compare the "leaked" event against your SDK configuration. This is the "Gap Analysis."

Reviewing your hooks

Most SDKs provide a beforeSend or beforeBreadcrumb hook. This is your first line of defense. If you find an email in a breadcrumb, look at your beforeBreadcrumb implementation. Are you actually scrubbing the data field of the breadcrumb, or just the message?

A common mistake is scrubbing the request object but forgetting that the same data might be duplicated in an extra field or a custom tag. Remember: tags are indexed and searchable, making them even more dangerous for PII leaks.

Server-side scrubbing vs. Client-side scrubbing

Many providers (like Sentry) offer server-side scrubbing rules. These are great as a safety net, but they are not a solution. By the time a server-side rule triggers, the PII has already left your infrastructure and travelled over the public internet to a third party. Your goal should always be to scrub at the edge—before the data leaves the client or your server.

Compare this found leak in your audit:

"tags": { "last_login_email": "ceo@victim-corp.com" }

Against your config:

Sentry.init({
  dsn: "...",
  denyUrls: [/password/],
  // Wait, I forgot to scrub tags!
});

The "Gap" here is that denyUrls only filters the URL of the crash, not the metadata attached to it. Every time you find a pattern in Phase 2, you should be able to trace it back to a missing line in your SDK initialization.

Phase 4: Instant Verification with the PII Detector

Writing custom regex and shell scripts is great for a deep dive, but it's slow for day-to-day development. If you want to quickly test a specific payload—perhaps one you're about to start sending from a new feature—you can use a dedicated tool.

We built the PII Detector tool specifically for this purpose. Instead of guessing how your SDK will handle a nested object, you can paste the JSON payload directly into the tool. It visualizes the "Scrubbing Map," showing you exactly which fields are flagged by standard compliance filters (GDPR, CCPA, HIPAA).

Visualizing the "Scrubbing Map"

The detector doesn't just look for keys like "email"; it runs a deep inspection of the values. It can distinguish between a UUID and a Credit Card number, or between a standard internal ID and a potential PII leak in a URL parameter. This is particularly useful for testing edge cases like:

• URL Parameters: Does https://example.com/reset?email=foo@bar.com get caught?
• Nested JSON: Does a stringified JSON blob inside a string field get inspected? (This is a common leak vector where one log level "double-encodes" data).
• Request Bodies: How does your tracker handle multipart/form-data?

Testing these scenarios in the PII Detector takes seconds, whereas setting up a reproduction in a staging environment to see what Sentry receives can take hours.

How to Implement Permanent Scrubber Rules

An audit is a snapshot in time. To ensure you don't regress, you need to move from one-time grepping to a hardened, permanent pipeline. Here's how to do it right.

Standardizing on Redaction Patterns

Don't use multiple different redaction strings like HIDDEN, [scrubbed], and ***. Pick one (we recommend [filtered] as it's the industry standard) and use it everywhere. This makes it much easier to write automated tests that scan your logs for the *absence* of PII. If you see anything that *isn't* [filtered] and matches a PII regex, you know you have a problem.

Moving Scrubbing to the "Edge"

If you are running on Cloudflare, you have a massive advantage. You can use a Cloudflare Worker as a "Scrubbing Proxy." Instead of sending data directly to your error tracker, send it to your own Worker. The Worker can then run a heavy-duty, recursive scrubbing algorithm on every event before forwarding it to the final destination. This ensures that even if a developer bypasses the client-side SDK settings, the PII is stripped before it leaves your network.

Here is a robust TypeScript example of a recursive scrubber you can use in a beforeSend hook or an Edge Worker:

const PII_KEYS = [/email/i, /password/i, /token/i, /secret/i, /auth/i, /card/i];

function scrub(obj: any): any {
  if (!obj || typeof obj !== 'object') return obj;

  if (Array.isArray(obj)) {
    return obj.map(scrub);
  }

  const scrubbed: any = {};
  for (const [key, value] of Object.entries(obj)) {
    if (PII_KEYS.some(regex => regex.test(key))) {
      scrubbed[key] = '[filtered]';
    } else if (typeof value === 'string') {
      // Check for email pattern in the string value itself
      if (value.includes('@') && /[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}/i.test(value)) {
        scrubbed[key] = '[filtered]';
      } else {
        scrubbed[key] = value;
      }
    } else {
      scrubbed[key] = scrub(value);
    }
  }
  return scrubbed;
}

// In your SDK init:
Sentry.init({
  beforeSend(event) {
    return scrub(event);
  }
});

This approach is "recursive," meaning it will dig into every nested object and array in your event. It also checks both the *keys* and the *values*, catching that last_login_email tag we found earlier.

For a deeper dive on setting up a fully Sentry-compatible pipeline that runs on your own infrastructure, check out our Sentry-compatible SDK guide.

Conclusion

PII redaction is not a feature; it's a constant process. The "1,000-event audit" should be a quarterly ritual for any team handling sensitive user data. By extracting raw data via the API, using shell tools to find leaks, and closing the gap in your SDK configuration, you transform your error tracker from a liability into a hardened tool.

Don't trust the "default" checkbox. Start your audit today: grab your last 1,000 events, run the grep hitlist, and if you find something suspicious, paste it into our PII Detector to see exactly what's getting through. It's better to find the leak yourself than to have a compliance auditor find it for you.