Quantitative data infrastructure & lakehouse architecture
Quant Researcher · Off-Cycle · Monaco · June 2025 – Present
During my off-cycle experience at Jukoï Capital in Monaco, I worked
on the design and development of a provider-agnostic financial data
infrastructure supporting quantitative research, market-data
processing, portfolio workflows, risk calculations and pricing
models.
The initial project focused on automating the ingestion,
transformation and normalization of market, macroeconomic and
alternative data. The resulting components were integrated into a
web-based financial application, connecting research workflows with
operational portfolio and risk-management tools.
I am now contributing to the design of a complete quantitative
lakehouse architecture intended to make the platform more
reproducible, observable and production-ready. The architecture
separates raw, enriched, staging, canonical and gold data layers,
while preserving source objects, provider vintages, immutable
releases and replayable transformations.
The target platform combines Ceph RGW for sovereign S3-compatible
object storage, Apache Iceberg for transactional analytical tables
and Lakekeeper as the central Iceberg REST catalog. Trino, Spark,
Polars and DuckDB provide shared SQL, distributed processing and
local iteration, while Dagster orchestrates partitioned data assets
and enforces data contracts.
A major focus of the architecture is financial reproducibility.
Critical datasets distinguish between event time, publication time
and verified availability time. Point-in-time feature construction
and anti-look-ahead tests ensure that backtests only use information
that would genuinely have been available at the time of each
historical decision.
The platform also introduces controlled AI-assisted workflows. Local
language models can support document extraction, data cleaning,
normalization, field mapping and pattern detection, but their
outputs remain restricted to enriched or staging layers until they
pass deterministic validation, quality checks and, where required,
human review.
The broader system includes lineage tracking, data-quality contracts,
observability, agent evaluation, vector-search synchronization and
disaster-recovery procedures. The objective is not to reproduce a
monolithic data platform, but to build a modular and auditable
foundation adapted to quantitative research and portfolio operations.
Challenge
Build a scalable and provider-agnostic financial data platform
capable of handling market data, macroeconomic data, alternative
data, portfolio processes and quantitative research while
preserving reproducibility, security and temporal correctness.
Solution
Designed a layered quantitative lakehouse based on Ceph RGW,
Apache Iceberg and Lakekeeper, with Dagster orchestration,
Trino and Spark processing, declarative data contracts,
OpenLineage tracking, Feast point-in-time features and controlled
MLflow-based AI evaluation.
Impact
Established a reusable foundation for certified datasets,
reproducible backtests, auditable research releases and more
reliable connections between data engineering, quantitative
research, trading and portfolio-management workflows.