Data Transformation Is the Hidden Bottleneck in Agent Pipelines

Every agent pipeline has the same dirty secret: the most expensive step isn't the LLM call. It's the data wrangling that happens before and after.

An agent pulls customer records from a CRM API that returns XML. The next skill expects JSON. The agent spends tokens converting formats in-context, gets the nesting wrong, retries, and burns through 4,000 tokens on a task that should have been a deterministic function call. Multiply this across a pipeline with five steps and three data sources, and you're spending more on format conversion than on the actual intelligence.

This is a skills problem. Data transformation is deterministic, well-specified, and exactly the kind of work that should be offloaded to purpose-built tools. Today we're adding three data primitives that eliminate this bottleneck.

The Three Data Primitives

1. Schema Inference from Sample Data

The first problem in any data pipeline is knowing what you're working with. APIs return undocumented JSON blobs. CSV files have inconsistent column types. Agents waste tokens asking "what fields does this have?" and "is this a number or a string?"

The Schema Inference Engine accepts 1-500 sample data objects and returns a JSON Schema (draft-07 or 2020-12) or TypeScript interface. It detects nullable fields, nested structures, array cardinality, and even auto-detects low-cardinality string fields as enums. A configurable confidence threshold handles messy real-world data where the same field contains mixed types.

At $0.001 per call, this replaces the 2,000+ tokens an agent typically spends inspecting and reasoning about data structure. The schema becomes a contract that downstream skills can validate against — no more "unexpected field type" errors three steps into a pipeline.

2. PII Detection and Anonymization

Privacy compliance in agent pipelines is usually an afterthought. An agent processes customer support tickets, extracts insights, and passes them to a summarization skill. The summaries contain email addresses, phone numbers, and occasionally SSNs that were embedded in ticket descriptions. By the time anyone notices, the PII has been sent to three external services.

The Data Anonymizer scans structured data for emails, phone numbers, SSNs, credit card numbers, IP addresses, names, and addresses. It supports five anonymization strategies: redact (remove entirely), hash (SHA-256), mask (preserve format with asterisks), generalize (city instead of full address), or synthetic (replace with realistic fake data). Custom regex patterns handle domain-specific identifiers.

At $0.002 per call, this is cheaper than a GDPR incident. Insert it as the first step in any pipeline that touches user data, and PII never leaves your trust boundary.

3. Universal Format Conversion

The agent economy runs on JSON, but the enterprise world runs on CSV, XML, YAML, and TOML. Every integration boundary is a format conversion. Agents handle this badly — they attempt format conversion in-context, hallucinate XML attributes, mishandle CSV escaping, and produce YAML with incorrect indentation.

The Universal Format Converter handles bidirectional conversion between JSON, CSV, XML, YAML, TOML, and NDJSON. It handles the edge cases that agents get wrong: nested object flattening for CSV, XML namespace preservation, configurable delimiters, and pretty-printing.

At $0.0005 per call, this is effectively free. A single failed retry due to bad format conversion costs more in tokens than a hundred converter calls.

The Cost Arithmetic

A typical data-heavy agent pipeline spends 3,000-8,000 tokens per run on data wrangling — format detection, conversion, validation, and PII concerns. At current token prices, that's $0.01-0.03 per run in pure overhead.

The three DataLens.ai skills replace all of that for $0.0035 combined:

That's a 3-9x cost reduction, with deterministic correctness instead of probabilistic token-based conversion.

Where Data Skills Fit in the Pipeline

Data transformation skills belong at pipeline boundaries — the entry and exit points where data crosses trust or format lines:

1. Ingestion: Infer schema from raw API response, anonymize PII, convert to canonical JSON
2. Inter-skill: Convert between formats when chaining skills with different input expectations
3. Egress: Convert results to the format the consuming system expects (CSV for spreadsheets, XML for legacy APIs)
4. Compliance checkpoints: Anonymize before passing data to external skills, re-identify after receiving results

The composition builder already supports multi-step pipelines. Adding data transformation skills as standard pipeline stages turns ad-hoc wrangling into a repeatable, auditable process.

Introducing DataLens.ai

DataLens.ai is the newest publisher on BluePages, focused exclusively on data enrichment and transformation for agent pipelines. All three skills are available now on the marketplace, with sub-200ms latency and 99.5%+ uptime.

Browse the full DataLens.ai skill set on BluePages or try them in the sandbox.