Engineering

Engineering

Methods, standards and architecture patterns I rely on, illustrated by systems I have actually built.

§ 01

Expertise, end to end.

What I take responsibility for, from source extract to decision.

01

Data pipelines and ETL

Layered pipelines (bronze, silver, gold) with input contracts, quarantine, idempotent reloads, execution logs and quality gates.
- Python
- pandas
- Jupyter
- Power Query
- Talend
02

Data warehousing and modelling

Kimball dimensional models: strict star schemas, conformed dimensions, audit dimensions and one-way relationships.
- SQL Server
- T-SQL
- MySQL
- Oracle SQL
03

Data quality and traceability

Versioned rule catalogues, row-level lineage back to the source line, and integrity certificates a third party can verify.
- Rule catalogue
- SHA-256
- Merkle trees (RFC 6962)
04

Semantic layer and BI

Power BI semantic models with centralised thresholds, explicit comparability rules, drill-through traceability and printable reports.
- Power BI
- DAX
- TMDL / PBIR
- Power BI Service
05

Metadata and governance

DDI documentation generated from a single reference, publication to NADA catalogues with role-based access to restricted files.
- DDI 2.5
- Dublin Core
- World Bank Metadata Editor
- NADA
06

Applied analytics

Explainable risk scoring, forensic tests and unsupervised anomaly detection, evaluated with explicit success criteria.
- scikit-learn
- Keras
- NumPy
- SciPy

§ 02

Principles

The rules I apply to every system, and what each one looks like in practice.

Reliability by design

A pipeline must give the same result twice and fail loudly when it cannot.

In practiceIdempotent reloads, blocking controls before any write, outputs never overwritten.

Data quality as a first-class concern

Quality rules are code: named, versioned, tested and reported, not hidden in a dashboard filter.

In practiceA 35-rule catalogue, quarantine of rejected rows, a quality report per execution.

Security and governance

Confidentiality is decided per field and per audience, before anything is published.

In practiceNamed fields kept out of public outputs, role-based access, rehearsed and reversible server upgrades.

Maintainability

One source of truth per decision, so a change happens in one place and propagates.

In practiceReference tables owned by the ETL, thresholds in one model folder, scripted and replayable changes.

Observability

Every number on a dashboard should lead back to the execution, file and line that produced it.

In practiceExecution identifiers, row hashes, lineage drill-through down to the source spreadsheet line.

Business alignment

Indicators follow the regulation and the users' questions, and say clearly when data is too thin to conclude.

In practiceRules mapped to the Public Procurement Code, minimum-volume rules before any year-on-year comparison.

§ 03

Reference architecture

The layered pattern behind my main platform. Each layer has one responsibility and one way to be checked.

Layers and their controls

Each layer is checked before the next one is written. Lineage is carried by a row hash; integrity by a Merkle certificate.

01 · Sources
- Extracts and filesInput contract
02 · Ingestion
- BronzeImmutable, hashed archive
03 · Validation
- Rule catalogueQuarantine, quality gates
04 · Transformation
- SilverConformed, row hash
05 · Storage
- Gold star schemaAudit dimension, execution log
06 · Semantic layer
- Model and metadataMeasures, DDI
07 · Decision
- Reports and catalogueTraceable to the line

§ 04

Standards and frameworks

Kimball lifecycleProject and warehouse design
CRISP-DMAnalytical and machine-learning work
DDI 2.5 · Dublin CoreStatistical metadata
RFC 6962Merkle-tree integrity proofs
DAMA-DMBOKData management vocabulary
WCAG 2.2Accessible reports and interfaces