A/B testing
Measuring whether a change really worked, with two random groups and an honestly read result.
What this course covers
The method for answering "did this change raise orders?" without being fooled by the season, payday or chance. You learn what an A/B test can and cannot answer, how to choose one primary metric, why a random split beats a before-and-after comparison, why small tests mislead, and how to read the interval of the difference B − A. Then hypothesis tests, sample size and power, and the pitfalls companies run into. Interactive widgets show how the number of visitors changes what you can conclude.
New to AI? Start with AI & data fundamentals.
Part of the career paths:Data analystData scientist
What you will be able to do
- Design an A/B test with one primary metric and guardrails
- Explain why a before-and-after comparison misleads
- Estimate how many visitors a test needs
- Read the interval of B − A and make an honest decision
Units and bites
- Unit 15 bitesAbout 15 min
A/B test essentials
What an A/B test answers, one primary metric, why randomise, why small tests mislead, and how to read the result.
- 3 min
What an A/B test answers
One change, two random groups, one period: an A/B test answers "did this change cause the difference?" and not much more.
- 3 min
Choosing one primary metric
Decide the one metric that settles the test before it starts, plus a few guardrails that must not get worse.
- 3 min
Why randomise
A "before vs after" comparison mixes the effect of the change with everything else that changed. A random split separates them.
- 3 min
Why small tests mislead
With few visitors, conversion rates swing so widely that a "winner" can appear by chance alone.
- 3 min
Reading the result
Read the interval of the difference B − A against 0: wholly above, wholly below, or spanning 0. Then ask whether the size is worth it.
- Unit 23 bitesAbout 9 min
Designing a test
Design, sample size and when to stop. Distributions and p-values are covered in the Statistics course.
- 3 min
Designing an A/B test
Randomisation unit, primary metric and guardrails before the test starts.
- 3 min
Sample size and power
How long a test has to run to detect a 0.5 percentage-point lift.
- 3 min
Peeking and stopping early
Stopping the first time p dips below 0.05 multiplies false alarms.
- Unit 34 bitesAbout 12 min
A/B test pitfalls
The things that make a well-designed test give the wrong answer anyway.
- 3 min
Novelty effects
Users click because something is new, not because it is better. How to spot it in week-by-week data.
- 3 min
Sample ratio mismatch
The plan was 50/50, the result 52/48. Why that is a sign something is broken, not chance.
- 3 min
Fishing in segments
"It did not win overall, but it won for Android users in Surabaya": why that is almost always chance.
- 3 min
When groups affect each other
On marketplaces and ride-hailing apps, groups A and B share the same sellers and drivers.
- Unit 44 bitesAbout 12 min
Running tests in a company
From one test to a habit: a written plan, guardrails, a test calendar and honest sharing of results.
- 3 min
Writing a test plan
Hypothesis, primary metric, sample size and stopping rule, written down before the test starts.
- 3 min
Guardrail metrics
Metrics that must not get worse: cancellations, complaints, app speed, margin.
- 3 min
Running tests side by side
When two tests can run at once, and avoiding tests during 11.11 or Lebaran.
- 3 min
Sharing test results
Writing up losing and unclear results as honestly as winners, so the company learns.
Questions about this course
Learning
How long does A/B testing take?
About 48 minutes in all: 4 units made of 16 bites of about three minutes each, and every bite ends with a quick check. You can do one bite a day or several in a row.
Do I need to know how to code?
Not to start. The foundations explain every idea in plain words first; where a course uses SQL or Python, it teaches it from the beginning, with a playground in the lesson.
Where should I start?
If you are new to AI, begin with AI & data fundamentals. If you know the job you want, follow its career path; A/B testing has a place in it.
When can I start learning?
Datamew is coming to iOS and Android soon. There is no release date or price announced yet: join the waitlist and we will tell you when the app is out.
Is it available in Bahasa Indonesia?
Yes. Datamew is written in English and Bahasa Indonesia.
More courses
- AI & data fundamentalsHow AI works, how to instruct it, and how to read numbers honestly.
- Data analystHow analysts think and work: from a business question to an answer people use.
- Data engineerBuilding pipelines that deliver data on time and correct: SQL, Python, orchestration and warehouses.
- Data scientistWhat data scientists do, and how to use LLMs in the work.
- Data preparationFrom a raw export to a table you can trust: tidy shape, missing values, duplicates and formats.
- Data visualisationCharts that answer a question honestly, from picking the type to dashboards and stories.
- SQL for analysisAsking the database directly: filtering, summarising, joining tables and window functions.
- StatisticsThe statistics every data role uses: summarising, reading distributions and measuring uncertainty.
- Supervised and unsupervised learningThe two ways machines learn from data: with answers, and without them.
- Product recommendationsHow "you might also like" works, the data behind it, and how to prove it pays.
- ForecastingForecast demand as a range, ready for Ramadan, 11.11 and 12.12.
- Customer segmentationSplitting customers into groups that differ, then treating each group differently.
- Social media analyticsRead Instagram, TikTok and YouTube numbers honestly, and use AI without being fooled.
- AI for finance reconciliationMatching bank statements to invoices: rules first, AI for the rest, people decide.
- AI for SEOHow search works in the age of AI answers, keyword research, Search Console, and writing with AI responsibly.
Read while you wait
Guides from the Datamew team, free on the web today.
How to Become a Data Analyst in Indonesia: Skills, Roadmap and Salary (2026)
To become a data analyst, learn spreadsheets, SQL, one BI tool such as Looker Studio or Power BI, basic statistics, and how to explain a finding in plain words. Most people can learn the basics in about six months part-time, then need two or three portfolio projects to get interviews. In Indonesia the average data analyst salary on Indeed is about Rp6.3 million a month.
8 min readHow to Become a Data Scientist in Indonesia: Skills, Roadmap and Salary (2026)
A data scientist uses Python, statistics and machine learning to predict what will happen and to test what works. Most people get there through data analysis first: learn SQL and statistics, then Python, experiments and machine learning, which takes roughly a year part-time. In Indonesia, data scientist salaries on Indeed average about Rp14.5 million a month.
8 min readBe first on the app
Leave your email and we will tell you the day Datamew lands on your phone. One email at launch, no spam.
- Coming soon onApp Store
- Coming soon onGoogle Play
- Use it now onthe web