Data Engineer
Building pipelines that deliver data on time and correct: SQL, Python, orchestration and warehouses.
What this track covers
The career track for aspiring data engineers. How data moves (ETL vs ELT, batch vs streaming, pipelines that are safe to re-run), SQL and data modelling, Python for pipelines, orchestration and scheduling as in Airflow or n8n, warehouses and lakehouses, and data quality and monitoring. The examples come from systems common in Indonesia: payment gateways, marketplaces and mobile apps. Best taken after AI & Data Fundamentals.
New to AI? Every track builds on AI & Data Fundamentals.
What you will be able to do
- Design pipelines that are idempotent and safe to re-run
- Model data with fact and dimension tables
- Schedule and orchestrate jobs with retries and alerts
- Test and monitor data quality
Units and bites
- Unit 14 bitesAbout 12 min
How data moves
Pipelines, ETL vs ELT, batch vs streaming, and pipelines that are safe to re-run.
- 3 min
What a data engineer does
Building and maintaining the data highway, so analysts and data scientists never have to think about it.
- 3 min
Anatomy of a pipeline
Trigger, extract, transform, load, destination — and when batch is enough.
- 3 min
ETL vs ELT
Tidy before loading, or load first and tidy in the warehouse — and why raw data is worth keeping.
- 3 min
Safe to re-run
Pipelines will fail now and then. What matters is that re-running one does not double the data.
- Unit 25 bitesAbout 15 min
SQL & data modelling
Fact and dimension tables, grain, and schemas that make analysts fast and numbers consistent.
- 3 min
App tables vs analytics tables
Why a schema that suits the app does not necessarily suit reporting.
- 3 min
Facts and dimensions
The events you measure and the context that describes them: the star schema.
- 3 min
Choosing the grain
What does one row represent? The most important decision in a fact table.
- 3 min
Joins that inflate numbers
Joining orders to items and then summing the order total: a classic mistake.
- 3 min
Changing dimensions
A customer moves city: do old reports change with them, or stay as they were?
- Unit 35 bitesAbout 15 min
Python for pipelines
Calling APIs, handling pagination and retries, and picking the right file format.
- 3 min
Calling an API
Requests, JSON responses and the status codes worth checking.
- 3 min
Pagination and rate limits
Fetching every page without breaking the requests-per-minute limit.
- 3 min
Retries and backoff
Trying again with growing pauses, and when to give up.
- 3 min
CSV, JSON and Parquet
Which format for exchange, which for analytics, and why types matter.
- 3 min
Secrets and config
API keys in the environment or a secrets manager, not in the code.
- Unit 45 bitesAbout 15 min
Orchestration & scheduling
Running many jobs in the right order with retries, alerts and backfills — as in Airflow or n8n.
- 3 min
What an orchestrator does
From a simple cron to a tool that knows the order, state and history of every job.
- 3 min
DAGs and dependencies
Jobs that wait for other jobs, and jobs that can run in parallel.
- 3 min
Schedules and backfills
Running a pipeline for missed dates without damaging the data.
- 3 min
Retries and alerts
How many times to retry, and who gets woken up if it still fails.
- 3 min
n8n vs Airflow
Visual flows for quick automation, code for big data pipelines.
- Unit 55 bitesAbout 15 min
Warehouses & lakehouses
Where analytics data lives: columnar storage, partitions, data layers and query cost.
- 3 min
OLTP vs OLAP
Databases that serve the app vs databases that answer questions.
- 3 min
Columnar storage
Why reading three columns of a 200-column table can be fast and cheap.
- 3 min
Lake, warehouse, lakehouse
Cheap files in object storage, tidy tables in a warehouse, and the blend of both.
- 3 min
Bronze, silver and gold layers
Raw, clean and ready-to-use data, and who each layer is for.
- 3 min
Partitioning and cost
The query that scans the whole table every five minutes, and the bill that swells with it.
- Unit 65 bitesAbout 15 min
Data quality & monitoring
Data tests, data contracts, monitoring, and how to handle a data incident.
- 3 min
Dimensions of data quality
Complete, unique, valid, consistent and timely: five things you can test.
- 3 min
Tests in the pipeline
not null, unique, accepted values and freshness, run every time data loads.
- 3 min
Data contracts
An agreement with the app team so a column change does not quietly break reports.
- 3 min
Monitoring and thresholds
Averages that hide breakage, and thresholds that actually raise the alarm.
- 3 min
Handling a data incident
Stop the damage, tell the data users, fix it, then prevent a repeat.
Questions about this track
Learning
How long does Data Engineer take?
About 87 minutes in all: 6 units made of 29 bites of about three minutes each, and every bite ends with a quick check. You can do one bite a day or several in a row.
Do I need to know how to code?
No. Every Datamew track is written for working professionals without a technical background. Where there is arithmetic, it is the kind you can do on a phone or in a spreadsheet.
Where should I start?
Everyone begins with AI & Data Fundamentals, which covers how AI works, how to write a good prompt and how to read data honestly. Then take Data Engineer for your role.
When can I start learning?
Datamew is coming to iOS and Android soon. There is no release date or price announced yet: join the waitlist and we will tell you when the app is out.
Is it available in Bahasa Indonesia?
Yes. Datamew is written in English and Bahasa Indonesia.
More tracks
- AI & Data FundamentalsHow AI works, how to instruct it, and how to read numbers honestly.
- Data AnalystFrom a business question to an answer people can use: spreadsheets, SQL, charts and dashboards.
- Data ScientistPredictions, experiments and models the business uses: Python, statistics and machine learning.
Read while you wait
Guides from the Datamew team, free on the web today.
How to Become a Data Analyst in Indonesia: Skills, Roadmap and Salary (2026)
To become a data analyst, learn spreadsheets, SQL, one BI tool such as Looker Studio or Power BI, basic statistics, and how to explain a finding in plain words. Most people can learn the basics in about six months part-time, then need two or three portfolio projects to get interviews. In Indonesia the average data analyst salary on Indeed is about Rp6.3 million a month.
8 min readHow to Become a Data Scientist in Indonesia: Skills, Roadmap and Salary (2026)
A data scientist uses Python, statistics and machine learning to predict what will happen and to test what works. Most people get there through data analysis first: learn SQL and statistics, then Python, experiments and machine learning, which takes roughly a year part-time. In Indonesia, data scientist salaries on Indeed average about Rp14.5 million a month.
8 min readBe first on the app
Leave your email and we will tell you the day Datamew lands on your phone. One email at launch, no spam.
- Coming soon onApp Store
- Coming soon onGoogle Play
- Use it now onthe web