04/DATA & INTEGRATIONS
Getting data from where it is to where it is useful.
Pipelines, reporting layers, third-party integrations and event-driven systems. The plumbing between the software you built, the services you pay for, and the numbers someone needs on Monday morning.
01/When you would call us
Four situations we hear a lot.
Reporting means someone exporting three CSVs on Friday and reconciling them by hand, and the numbers still disagree.
You need to take payments, or you already do and the webhook handling has started double-charging people occasionally.
A partner API drops out for ten minutes and your system loses the records that arrived during it, permanently.
Data lives in five systems and no single one of them can answer a question your board asks every month.
02/What we do
The work, itemised.
- Data pipelines
- Scheduled and event-driven pipelines that pull from sources, validate, transform and land data where it is queryable — with failures that announce themselves rather than producing quiet gaps.
- Reporting layers
- A modelled layer built for the questions people actually ask, so a metric has one definition and one number instead of three depending on who exported it.
- Payment integrations
- Stripe and similar providers, with the parts that cause real damage handled properly: idempotent webhook processing, signature verification, retries, refunds and reconciliation against your own records.
- Third-party integrations
- CRMs, ERPs, messaging platforms and partner APIs, wrapped so their outages become a queue and a retry rather than an incident on your side.
- Event-driven architecture
- Decoupling services with SQS, SNS, EventBridge and Step Functions, including the unglamorous parts — dead-letter queues, replay, ordering guarantees and what happens on a duplicate.
- Data migration
- Moving data between systems or schemas with validation on both sides, a dry run against production-shaped data, and a reconciliation report proving nothing was lost.
03/How we approach it
The order matters more than the list.
Trace the data to its source
Before anything is built we follow each field back to where it is created and find out who owns it. Most reporting disagreements turn out to be two systems defining one word differently.
Agree the definitions in writing
What counts as an active customer, when revenue is recognised, which timezone a day starts in. Boring to write down and the reason the numbers reconcile afterwards.
Build the unhappy path first
Duplicate messages, malformed rows, a source that returns an error page instead of JSON. Handling these first is why the pipeline still works in month six.
Backfill and reconcile
History gets loaded, then checked against the source system record by record. You see the reconciliation report, including anything that did not match and why.
Instrument and schedule
Freshness checks, row-count expectations and alerts on a pipeline that ran without error but produced nothing — the failure mode that otherwise goes unnoticed for weeks.
04/Technology
What this pillar is built with.
- Languages
- Python
- TypeScript
- SQL
- Pipelines
- Lambda
- Step Functions
- EventBridge Scheduler
- Glue
- Messaging
- SQS
- SNS
- EventBridge
- Dead-letter queues
- Storage
- S3
- RDS
- Aurora
- DynamoDB
- Athena
- Integrations
- Stripe
- REST and GraphQL APIs
- Webhooks
- SFTP
05/What you get
Things you can point at.
- Pipelines that run on a schedule or an event, with retries and dead-letter handling
- A written data dictionary: every field, its source, its owner and its definition
- A reconciliation report showing source and destination agree
- Webhook handling that is idempotent and signature-verified
- Alerts for freshness and volume, not only for outright errors
- A replay procedure for reprocessing a window after a fix
Related services and solutions
06/Questions
Data & Integrations, asked about.
01Do we need a data warehouse?
Frequently not. If your data fits comfortably in Postgres and the queries return quickly, a modelled reporting schema in the database you already run is cheaper to build and cheaper to operate. We will say so when that is the case.
02Can you integrate with a system that has no API?
Usually, through whatever it does expose — scheduled exports, SFTP drops, database replicas, or a headless session as a last resort. We will be direct about how fragile each option is before you commit to one.
03How do you handle a third-party API going down?
Requests go through a queue with retries and backoff, so an outage becomes a delay rather than lost data. What cannot be retried lands in a dead-letter queue with enough context to replay it once the provider recovers.
04Can you fix a pipeline someone else built?
Yes. That starts with a read-through and a written list of where it can silently lose or duplicate data. Those failure modes get fixed before anything is added on top.
Describe the problem.
A few lines is enough. You will get a reply from an engineer within one business day, and it will address what you actually asked.