Survival Analysis: Kaplan-Meier Estimator
Why ordinary averages fail with time-to-event data
Suppose you want to know the average time until a customer cancels a subscription, a patient relapses, or a machine part fails. The obvious move — average the observed durations — breaks down the moment some subjects haven't experienced the event yet by the time you analyze the data. A customer who signed up last week and hasn't churned isn't a churn time of zero, and it isn't infinite either. This is censoring: we know the subject survived at least this long, but we don't know the true event time.
Survival analysis is the branch of statistics built to handle exactly this. It models the time until an event occurs, while correctly using the partial information in censored observations instead of discarding them.
The survival function
The central object is the survival function $S(t) = P(T > t)$: the probability that the event hasn't happened by time $t$. It starts at $S(0) = 1$ and decreases monotonically toward 0 (or toward some floor, if not everyone eventually experiences the event).
Related is the hazard function $h(t)$, the instantaneous event rate at time $t$ given survival up to $t$. The two are linked: $S(t) = \exp\left(-\int_0^t h(u)\,du\right)$. Hazard is a rate (events per unit time among survivors), while $S(t)$ is a cumulative probability — this distinction matters a lot once you move to the Cox model.
The Kaplan-Meier estimator
The Kaplan-Meier (KM) estimator, introduced by Edward Kaplan and Paul Meier in 1958, is the nonparametric way to estimate $S(t)$ directly from data without assuming any particular distribution of event times.
The idea: order the distinct event times $t_1 < t_2 < \dots < t_k$. At each $t_i$, let $d_i$ be the number of events at that time, and $n_i$ the number of subjects still at risk just before $t_i$ (not yet censored, not yet experienced the event). The probability of surviving past $t_i$, given survival up to $t_i$, is estimated as $1 - d_i/n_i$. The KM estimator multiplies these conditional survival probabilities:
$\hat{S}(t) = \prod_{i: t_i \le t} \left(1 - \frac{d_i}{n_i}\right)$
This product form is why KM curves look like step functions: $\hat{S}(t)$ drops at each observed event time and stays flat in between. Censored subjects simply leave the risk set at their censoring time without contributing a drop — they inform $n_i$ for earlier times but never force a decrease themselves.
Subjects and times (+ means censored, no + means event occurred):
2, 3, 5+, 6, 8, 8, 10+
Risk set shrinks as time passes; only true events (not censored) drop S(t):
t=2: n=7, d=1 -> factor = 1 - 1/7 = 0.857 S(2) = 0.857
t=3: n=6, d=1 -> factor = 1 - 1/6 = 0.833 S(3) = 0.714
t=5: censored, no drop, but n shrinks to 4 for next step
t=6: n=4, d=1 -> factor = 1 - 1/4 = 0.750 S(6) = 0.536
t=8: n=3, d=2 -> factor = 1 - 2/3 = 0.333 S(8) = 0.179
t=10: censored, curve stays flat at 0.179 (undefined past last event)Censoring is information, not missing data
import pandas as pd
from lifelines import KaplanMeierFitter
durations = [2, 3, 5, 6, 8, 8, 10]
event_observed = [1, 1, 0, 1, 1, 1, 0] # 1 = event, 0 = censored
kmf = KaplanMeierFitter()
kmf.fit(durations, event_observed=event_observed, label="churn")
print(kmf.survival_function_)
print("Median survival time:", kmf.median_survival_time_)
# Compare two groups (e.g. control vs. treatment) with a log-rank test
from lifelines.statistics import logrank_test
result = logrank_test(
durations_A, durations_B,
event_observed_A=events_A, event_observed_B=events_B,
)
print("p-value:", result.p_value)Comparing groups: the log-rank test
A KM curve per group is useful visually, but to test whether two survival curves differ statistically, use the log-rank test. It compares, at every observed event time across both groups, the observed number of events in each group against the number expected under the null hypothesis that the groups share the same hazard. Summing a chi-squared-like statistic across all event times yields a single p-value for 'do these curves differ at all.' It's the survival-analysis analogue of a t-test, but it makes no assumption about the shape of the survival curves — only that the ratio of hazards is roughly constant over time (the same proportional-hazards assumption that underlies the Cox model).
KM assumes censoring is non-informative
What's next?
Kaplan-Meier gives you a descriptive, nonparametric picture of survival for one group or a handful of groups. But it can't tell you how a continuous covariate — like age, dosage, or usage frequency — shifts the hazard, and comparing more than a couple of groups with log-rank tests quickly gets unwieldy. The natural next step is a regression model for survival data that lets hazard depend on multiple covariates simultaneously.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.