Daniel Kim Advisory memo Decision Call · USD 220
The activation metric everyone reports is measuring the wrong thing
60 minute call, 14 April. Reviewed and signed off by Daniel Kim on 16 April.
You asked why activation is flat at 34 percent while every qualitative signal says onboarding improved. It is flat because the metric counts an event that fires on a page load, so it has been measuring visits to a screen rather than anything a user did.
Activation is defined as fired the setup_complete event, and that event fires when the setup page renders, not when setup is completed. On a corrected definition, activation is 21 percent and has risen from 16 percent since the onboarding change in January. The work did what you thought; the instrument did not.
The question
What was asked, and what was reviewed
As you put it: "We rebuilt onboarding, support tickets fell, sales say it is better, and activation has not moved. What am I missing?"
| Reviewed | Source |
|---|---|
| Activation metric definition and query | Supplied |
| The setup_complete event implementation | Read on the call |
| 14 months of activation by weekly cohort | Supplied |
| Onboarding funnel, step by step | Supplied |
| Support ticket volume on setup themes | Supplied |
| Retention by activation status | Supplied. Clause 3 |
The defect
What the event actually fires on
SetupPage.tsx, as deployed
useEffect(() => {
analytics.track('setup_complete', { userId, plan });
}, []); // fires on mount
// The intent was to fire this in the submit handler.
// It was moved into a mount effect in Feb 2026 during a
// refactor, and nothing failed, so nothing was noticed.| What the name implies | What it counts |
|---|---|
| A user completed setup | A user loaded the setup page |
| An intentional action | A render |
| Once per user, at completion | Once per mount, including on a back navigation |
| A meaningful milestone | Reaching the screen, which is not the hard part |
This is why the number is both too high and insensitive: it captures everyone who arrives at setup, which the onboarding change did not affect, and misses everyone who completes it, which is what the change improved.
Better
What activation should mean here
The useful definition of activation is the earliest action that predicts retention. You already have the data to find it, and I ran it on the call.
| Candidate definition | Share of users | Day 90 retention if done | If not |
|---|---|---|---|
| Loaded the setup page | 34 percent | 39 percent | 36 percent |
| Completed setup | 21 percent | 58 percent | 31 percent |
| Invited a second user | 17 percent | 74 percent | 28 percent |
| Created 3 records | 24 percent | 61 percent | 30 percent |
| Returned in week 2 | 29 percent | 66 percent | 27 percent |
The current definition separates retention by 3 points, which is to say it separates nothing. Inviting a second user separates it by 46 points and happens early enough to act on. That is your activation metric.
How
Changing it without losing the history
- Fix the event first, in the submit handler, and rename it setup_submitted so nobody confuses old and new data later.
- Backfill the second-user definition from source. You have the invite records, so the whole 14 months can be recomputed rather than starting from today.
- Publish both series side by side for one quarter, with a note. Changing a headline metric silently is how people stop trusting the data team.
- Tell the people who own the January work what it actually achieved, before you tell the board.
- Write down the definition, in words, next to the number on every dashboard. Most metric drift starts with a definition that lives only in a query.