Senior Data Engineer · Jakarta

I build the pipelines that carry a company's data, from source system to boardroom.

4 years building production lakehouse platforms on AWS and Databricks for enterprises in retail, healthcare, telco and automotive.

Engraved illustration of a striped hot air balloon rising above an open field, tethered by a rope, with small figures in 19th century dress watching
Plate I. The ascent: careful groundwork, then lift. Fig. 01

Senior data engineer at Insignia, leading delivery across concurrent enterprise engagements and owning Databricks architecture standards company-wide.

My core is AWS lakehouse architecture (Glue/Spark, Iceberg, Lake Formation), Databricks (Unity Catalog, Delta Sharing, Lakeflow), Airflow and CDC pipelines. Before that I worked closer to the ML side: LLM reporting, NER models and vector search, which still shapes how I design data for AI workloads.

Cloud and platform

AWS: Glue/Spark, S3, AthenaLake Formation, Lambda, DMSDatabricks, Snowflake

Processing

Spark, Airflow, IcebergKafka, KinesisCDC (DMS), Airbyte

Data and ops

Python, SQLPostgreSQL, MongoDB, RedisGitHub Actions, Docker, MLflow

02

Selected projects

Enterprise client work. Client names withheld; happy to walk through details in conversation.

I.

Near real-time ingestion platform

Telco infrastructure

Architected an ingestion platform processing 10 to 15 GB a day from heterogeneous sources: SAP, RDBMS, POS, SFTP and flat files, landing into a governed lakehouse.

AWS Glue · DMS · Iceberg · Airflow

II.

7 TB historical migration

Automotive conglomerate

Led the migration of 7 TB of historical data: migration strategy, a validation framework to prove parity, and the cutover plan.

Migration · Validation · Cutover

III.

Data modernization program

Enterprise client

43 production pipelines and 30 GB monthly throughput, rebuilt on standardized ingestion patterns and medallion-layer conventions that were then reused across other projects.

Databricks · Unity Catalog · Lakeflow

IV.

Lakehouse lift-and-shift

Healthcare

Built and delivered a 30-pipeline migration into a lakehouse, covering data standardization, metadata centralization and schema governance.

AWS · Lake Formation · Iceberg

V.

CDC and streaming patterns

Unified Data Platform standard

Reusable CDC and streaming ingestion from Oracle, SQL Server and SFTP into an Iceberg lakehouse on AWS, shipped through GitHub Actions CI/CD and adopted as the team's default pattern.

Glue · DMS · Kinesis · Athena

VI.

LLM automated reporting

NTX Solusi Teknologi

Production LLM solution that cut client report creation time by 90%, backed by NER models at 95% accuracy and an HNSW vector search for retrieval.

LangChain · MLflow · FastAPI

Jun 2025 to now

Senior Data Engineer · Insignia, Jakarta

Promoted after 3 months. Leads delivery across 4 concurrent enterprise engagements with a team of 7. Internal Databricks authority: reviewed and approved 4 project architectures. Set the team's engineering SOPs and led pre-sales that helped win two major enterprise accounts.

Apr to Jun 2025

Mid. Data Engineer · Insignia, Jakarta

Owned end-to-end delivery of platform modernization work. Defined architecture standards for client Unified Data Platforms, built AWS lakehouses on Iceberg, and introduced CI/CD for pipeline deployments.

Dec 2023 to Mar 2025

Data Engineer · NTX Solusi Teknologi, Jakarta

ETL/CDC pipelines and a MinIO data lake orchestrated with Prefect. Great Expectations validation caught 95% of anomalies before ingestion. Shipped an LLM reporting product, ran monitoring at 98% uptime, and rolled out DataHub and Amundsen for discovery.

Mar to Jun 2023

AI Engineer · PERGA, Hamburg (remote)

NER and document classification based on LayoutLM, reaching 92% accuracy at the first checkpoint.

2019 to 2023

B.Sc. Information Systems · Universitas Airlangga

Thesis: session-based recommendation system using gated graph neural networks and attention for e-commerce. Top 5 AI Startup Final Project, Orbit Future Academy.

Certifications

AWS Certified Data Engineer, 2025 Databricks Certified Data Engineer Associate

04 · Contact

Have data that needs somewhere to go? Let's talk.

luthfirdty@gmail.com