Ange-Ghislain KOUAMEData Engineer
FR
Prototype

Explainable Risk Scoring and Unsupervised Anomaly Detection

An auditable scoring grid, forensic tests and a Keras autoencoder benchmarked against FAMD, with success criteria fixed before computation and limits reported as found.

Role
Designer and engineer of the analytical module
Period
Oct 2026
Status
Prototype
Context
Analytical extension of the procurement data platform · awaiting business validation
  • 13signals in three families in the scoring grid
  • 0.987median rank correlation across 2,000 perturbed weight grids
  • 6.4×lower reconstruction error, autoencoder vs FAMD
  • p = 0.66DeLong test: no significant targeting gain over FAMD
  • 11 / 11pre-registered success criteria met by the reference model
  • 35tests validating the NumPy fallback engine against scikit-learn and SciPy

Engineering figures describe the system, not its business findings, which remain confidential.

Analytical flow

Both methods start from the certified dataset and write to the warehouse through an audit dimension; Power BI reads current models only.

  1. 01 · Certified input

    • Certified silver datasetHash and Merkle root checked

  2. 02 · Expert signals

    • Scoring grid13 signals, capped at 100
    • Forensic testsThresholds, Benford, splitting

  3. 03 · Unsupervised learning

    • AutoencoderKeras 3 on PyTorch
    • FAMD baseline6 axes
    • TypologyWard and k-means, 4 classes

  4. 04 · Evaluation

    • Criteria CR-01 to CR-12ROC, DeLong, stability

  5. 05 · Deployment

    • Audit dimension and factsOne current model
    • Priorities and atypicality pages'Why' text per contract

01Executive summary

Auditors cannot review every contract. I built two complementary ways to prioritise them on top of the certified dataset: an explainable expert score, and an unsupervised model that flags contracts that look unusual.

The scoring grid combines 13 signals in three families (procedure regularity, financial reliability, data-entry quality) and explains each score in plain words. The unsupervised module, run in Keras, compares an autoencoder with a linear baseline (FAMD) under criteria fixed before any computation. The honest result: the autoencoder reconstructs the data 6.4 times better, but does not target high-priority contracts significantly better than FAMD.

02Objectives and constraints

  • Rank contracts for review with a score anyone can read: points per signal, amount at stake, and what to check.
  • Never use a person or an administrative section as a score input.
  • Find unusual contracts that no rule in the catalogue flags.
  • Decide deployment with criteria written before the computation, not after.

03Pipeline and processing

  • Forensic tests: threshold-bunching tests with exact binomial tests and a Bonferroni correction over 16 thresholds (11 used as controls), procedure use above its legal ceiling, presumed contract splitting, and Benford first- and two-digit tests with Nigrini's mean absolute deviation.
  • Scoring grid 2026.10-v1: score capped at 100, four score classes, four stake classes, and a P1 to P4 priority matrix.
  • Robustness: 2,000 grids with weights drawn between 0.75 and 1.25 times their value, plus a stress test between 0.5 and 1.5.
  • Funnel plots with exact binomial limits to compare administrative units fairly despite very different volumes.
  • Unsupervised module structured along the six CRISP-DM phases: feature preparation in the FAMD style, autoencoder, atypicality degree, Ward and k-means typology, classifier and decision-tree rules for new contracts, and drift monitoring.

04Data quality and validation

  • Twelve success criteria (CR-01 to CR-12) fixed before computation, five of them blocking: targeting above chance, reconstruction at least equal to FAMD, partition stability, classifier accuracy and identical reload.
  • Paired DeLong tests to compare ROC areas on the same contracts, instead of reading raw AUC differences as wins.
  • Two engines compared: Keras and an independent NumPy engine; the core of the typology survives the change of engine, the small classes do not.
  • Reproducibility: two runs give byte-identical outputs on the same machine.

05Technology stack

  • Python
  • pandas
  • NumPy
  • scikit-learn
  • SciPy
  • Keras 3
  • PyTorch
  • SQL Server
  • Power BI
  • CRISP-DM

06Engineering decisions

  1. Pre-registered success criteria

    Choice
    Twelve criteria written in the notebook before any training run.
    Why
    Prevents choosing the metric that flatters the model after seeing the results.

  2. A linear baseline, always

    Choice
    FAMD computed on the same features and compared with paired DeLong tests.
    Why
    Without a baseline, a neural network's complexity cannot be justified; here it was not justified for targeting.

  3. No person in the model

    Choice
    Agents and controllers excluded from features; joined only in the report, for filtering.
    Why
    The model describes contracts, not people, which limits misuse and keeps the analysis defensible.

07Technical challenges

  • Windows application control blocked the PyTorch libraries on the workstation. Rather than wait, I wrote an 823-line NumPy engine exposing the same interfaces (dense networks with Adam, PCA, k-means, Ward, CART, silhouette, chi-square) and validated it with 35 tests against scikit-learn and SciPy.
  • A NumPy boolean silently broke a criteria count ('10 of 11' instead of '11 of 11'); found during review, fixed, and the warehouse description corrected in place.
  • Explanations had to stay readable for non-specialists: amounts under one million written in francs, singular and plural handled, a 'why atypical' sentence per contract.

08Results and deliverables

  • A prioritised review list with a plain-language explanation and suggested check per contract.
  • A Power BI page and a two-page printable report on classes and atypical contracts, written for readers who do not practise data science, with a 13-term glossary.
  • A typology classifier with 99.6 % accuracy and 0.978 macro F1, and a five-rule decision tree faithful to 96.3 %.
  • A finding worth reporting as such: the typology mostly reflects the contract type, so it is a consistency check rather than a discovery; the real contribution of the unsupervised method is atypicality.

09Limitations

  • No reviewed decisions yet: the grid is expert-weighted and not calibrated on outcomes.
  • The list of atypical contracts depends on training as much as on the random seed; it should be read as a ranking, and contracts flagged by both engines treated as the most reliable leads.
  • An atypical contract is unusual, not irregular: every output says so explicitly.

10Future improvements

  • Recalibrate the grid by shrinkage towards expert points once at least 20 confirmed or justified decisions are available.
  • Train a supervised model on reviewed decisions and compare it with the current score.
  • Evaluate Isolation Forest as a third anomaly-detection baseline.