Home on Hâtvalues

Not All Skew Is Suspicious - How to Avoid Mistaking Signal for Outliers

Fri, 02 May 2025 00:00:00 +0000

Not All Skew Is Suspicious #

Introduction #

You’re exploring your data. You see a long tail or a cluster of extreme values. Your instinct? Flag them as outliers. Trim the noise. Clean the dataset.

But here’s the thing: not all skewed data is dirty. Skew can carry meaningful structure about the real-world process you’re modeling.

In this post, we’ll break down how to distinguish expected skew from true anomalies—and avoid costly mistakes in your EDA pipeline.

1. Understand What’s Generating the Data #

Many real-world processes naturally produce skewed distributions. For example:

Counts of events → Poisson (e.g. support tickets, transactions)
Durations and monetary values → Log-normal (e.g. customer LTV, time-on-site)
Multiplicative processes → Heavy right tails (e.g. viral growth)

If you trim here, you’re cutting real signal.

2. Leverage Domain Knowledge #

Outliers are context-dependent. $5,000 in revenue may be huge for one team, normal for another. Talk to SMEs. Ask:

“What values would surprise you?”
“Is this variability typical?”

3. Compare Empirical Distributions to Theory #

Overlay histograms or Q-Q plots against known distributions:

Smooth tails → expected skew
Sudden jumps or spikes → investigate further

4. Use Transformations as a Diagnostic Tool #

Try log, square root, or Box-Cox:

If it normalizes: it was likely an expected skew.
If values remain extreme: now you may have a real outlier.

5. Don’t Rely on Arbitrary Cutoffs #

The “1.5×IQR” rule isn’t gospel. It assumes symmetric distributions.

Instead, try:

Median absolute deviation (MAD)
Winsorization for modeling robustness (if justified)

6. Analyze in Context #

Break down the data:

By time
By customer segment
By product category

Outliers often vanish in the right slice of the data.

7. Use ML or Statistical Tools for Support—Not Final Judgment #

Isolation Forests
DBSCAN
Cook’s Distance
Local Outlier Factor

These help—but you still need to interpret the results.

Conclusion: Respect the Tail #

What looks like an outlier might be the tip of a trend. EDA is about understanding—not just cleaning.

Before trimming:

Ask what the skew tells you
Transform, visualize, segment
Model accordingly

👉 Skew isn’t always a sign of a problem. Sometimes it’s the story your data is trying to tell.

Optimize Like a Pro With LSD

Thu, 10 Apr 2025 00:00:00 +0000

Introduction #

Start-ups often need to move faster than traditional A/B testing best practices allow. Typically, A/B tests need a couple of weeks to gather enough data, sometimes more. When multiple improvements are ready to ship, waiting to test them one at a time can mean lost momentum or missed opportunities. Enter the Latin Square Design (LSD), a brilliant example of working smarter instead of harder. As a result of using LSD, your estimate of the treatment effect has significant sources of noise removed, which means:

Fewer users required to detect a real effect with sufficient power.
Random fluctuations in behavior across known sources of variance don’t affect your results.
You reduce the risk of confounding effects muddying your conclusions.

In this post, I’ll walk through a recent scenario where the LSD turned out to be the perfect choice.

Overview of Latin Square Designs #

LSD is a statistical power tool that lets you test multiple treatments while controlling for two nuisance factors, all without blowing up your sample size or adding noise to your results. It cleanly separates the treatment effect from two known sources of variability. What’s striking is that thanks to the way it’s set up, LSD drastically reduces the number of test conditions you need to run while maintaining statistical power!

Instead of testing every possible combination (which would require $t \times r \times c$ runs for $t$ treatments and $c, r$ levels of each factor), the LSD only needs $t \times t$. That’s a major win-win.

How does it work? The secret lies in its structure. Each treatment is assigned in such a way that it appears exactly once in every row and column of a square layout — ensuring a balanced representation across both blocking factors. Here’s what that looks like for three factor levels:

	x	y	z
I	A	B	C
II	B	C	A
III	C	A	B

The catch? Latin Square designs require the number of treatments to match the number of levels in each blocking factor to form the square. That symmetry requirement is what makes LSDs elegant — but also a bit rigid. If your real-world variables don’t fit neatly into this shape, more flexible alternatives like randomized block designs or factorial designs might be a better fit.

Just Enough Theory #

As with all my posts, I like to cover just ennough theory to allow readers to implement things for themselves. Fortunately, that’s not very much to grasp the basics of LSD.

Mathematically, the model looks like this:

$$ Y_{\textit{ijk}} = \mu + \alpha_i + \beta_j + \tau_k + \epsilon_{\textit{ijk}} $$ Where:

$Y_{\textit{ijk}}$ is the observed response (e.g., conversion)
$\mu$ is the overall mean
$\alpha_i$ is the row effect
$\beta_j$ is the column effect
$\tau_k$ is the treatment effect
$\epsilon_{\textit{ijk}}$ is the random error

By explicitly controlling for two known sources of variation (the blocking factors) before estimating the treatment effect, the total (T) variance in the observed data is partitioned into several components: row (r), column (c), treatment (t), and residual error (e). These components are easily discovered by creating a standard ANOVA table and show up as the sum of squares (SS):

$$ \text{SS}_T = \text{SS}_r + \text{SS}_c + \text{SS}_t + \text{SS}_e $$

You can also determine the relative efficiency (RE) of the design compared to a standard A/B test, one that would use a Complete Randomized Design (CRD) instead for the two blocking factors by the following formula:

$$ \text{RE} = \frac{\text{SS}_e^{\text{CRD}} \times \textit{df}^{\text{Blocked}}}{\text{SS}_e^{\text{Blocked}} \times \textit{df}^{\text{CRD}}} $$

where $\text{SS}_e^{\text{CRD}}$ is estimated from the unpartitioned sums of squares:

$$ \text{SS}_e^{\text{CRD}} = \text{SS}_r^{\text{Blocked}} + \text{SS}_c^{\text{Blocked}} + \text{SS}_e^{\text{Blocked}} $$

and the residual degrees of freedom (df) for the CRD is the df for the LSD plus the df for the factor components:

$$ \textit{df}^{\text{CRD}} = \textit{df}_r^{\text{Blocked}} + \textit{df}_c^{\text{Blocked}} + \textit{df}_e^{\text{Blocked}} $$

When $\text{RE} > 1.0$, the design is more efficient than the equivalent CRD (or standard A/B test). The greater the blocking effect (i.e., the larger the difference between the blocks), the more efficient the design becomes.

If you find RE is less than one, a common cause is that there is not much between block variability. The statistical effects of the blocking factors is weak and doesn’t need controlling. In this case, df are consumed for no benefit. Another issue could be that the number of replicates is too low, increasing variance estimates. For these reasons, it is always important to check the RE.

Scenario: Testing a New Checkout Flow #

Let’s now discuss a real-world example. Recently, we had two related updates to the shopping cart experience:

A one-click payment flow — based on user feedback about usability.
A change allowing users to edit their cart later in the checkout process — aimed at reducing drop-off after cart review.

Both features were ready to trial at the same time, and both were designed to improve completions after users initiated checkout. Rather than run two sequential tests, we decided to evaluate both changes in parallel by comparing three cart flows: the current version, one with the one-click payment, and one with the enhanced editing feature.

Experiment Design #

Our customer data platform offers several ways to define audiences. We know from previous tests that different audience segments show distinct purchasing behavior. So, for this experiment, we selected a segmentation that gives us a relatively even three-way split that we know captures variability in shopping cart completion: Regulars (frequent return customers), Responders (those responsive to email marketing and offers), and Others (everyone else).

Geography is another important source of variability in shopping habits. To reduce this noise, we limit experiments to a few cities with consistently strong and stable sales. This helps account for geographic and seasonal effects while minimizing interference from local marketing activity.

Running a full factorial design would require a larger sample size and more operational complexity. Given two treatments and two blocking factors, we chose the LSD as a more practical alternative. It allows us to control for both audience segment and city while comparing the three layouts more efficiently.

Latin Square Randomization #

As with any controlled experiment, we use randomization to assign treatments across blocks and units. This helps eliminate selection bias and ensures that any observed effects are due to the treatments rather than systematic differences in how groups were formed. In R, this is straightforward to implement using the agricolae package (De Mendiburu, Felipe - 2009), which provides convenient functions for generating randomized Latin Square layouts and other experimental designs.

cart_flow <- c("Control", "Easy Payment", "Easy Edit")
outdesign = design.lsd(cart_flow, seed = 232)
lsd <- outdesign$book[, 2:4]
levels(lsd$row) <- c("Regular", "Responder", "Other")
levels(lsd$col) <- c("Hamburg", "Istanbul", "Kraków")
names(lsd) <- c("audience", "city", "cart_flow")

square_layout <- lsd %>% pivot_wider(
  names_from = audience,
  values_from = cart_flow
)
kable(square_layout)

city	Regular	Responder	Other
Hamburg	Control	Easy Edit	Easy Payment
Istanbul	Easy Payment	Control	Easy Edit
Kraków	Easy Edit	Easy Payment	Control

Setting an Appropriate MDE #

Our formula for determining the minimum detectable effect (MDE) for each shopping cart experiment was agreed by a working group formed of the product manager, marketing manager and senior data scientist. We focus on the expected increase in the cart completion rate (CCR) by setting a realistic target cart completion rate (TCCR). The CCR is simply the mean proportion of daily completed carts (DCC) from the daily total carts (DTC).

$$ \text{MDE} = \text{TCCR} - \text{CCR},\ \text{CCR} = \frac{1}{N}\sum_{n=1}^{N} \frac{\text{DCC}_n}{\text{DTC}_n} $$

We try to set the TCCR taking into account the average daily cart value (ADCV). This helps to ensure that the expected increase in revenue (EIR) will cover the cost of the work:

$$ \text{EIR} = \text{MDE} \times \text{DTCV} \times \text{D} $$ where:

$\text{DTCV}$ is the daily total cart value, and
$\text{D}$ is the number of days we want to account for.

To plug in some real values, we pull the daily numbers for a month prior to the experiment for the three cities in the study. From this we get a mean CCR of 0.6071. We set the MDE to 0.055 for a TCCR of 0.6621. To determine the expected revenue increase for one month, should we achieve this TCCR, we multiply this MDE by the current average daily revenue €939.01 by the average days per month 30.4375 (we use this number to smooth out the uneven calendar). This gives a monthly EIR of €1571.96 from just these three cities. Extrapolating to rest of world, the increase revenue is more than adequate to cover this mini-project.

Power Calculation #

Once we have agreed the MDE, we can calculate the number of replicates required to have confidence in the result. In this case, a replica means running the same Latin Square for a complete day.

We need to calculate Cohen’s $f$, the standardised effect size of a difference in proportions between groups. The formula for this is:

$$ f = \sqrt {\frac{\sum^K_{k=1}(\pi_k - \bar{\pi})^2}{K\sigma^2}} $$ where $\pi_k$ is the proportion $\frac{successes}{trials}$ for the $k^{th}$ treatment group and $K$ is the number of treatment groups. The numerator is therefore the between groups sum of squares, and the denominator is the within groups variance, estimated in advance from the data for the previous month on the current cart version.

For this experiment, we can plug in $K = 3$, $\pi_1 = 0.6071$, which is the control group CCR (prior). Let $\pi_2 = 0.6621$ (our TCCR) and $\pi_3$ can be ignored for this calculation e.g. set a neutral value such that $\pi_3 = \text{Abs}(\frac{\pi_1 - \pi_2}{2}) = \bar{\pi}$. Finally, $\sigma^2 = 0.0061$ (prior).

This gives us a value of $f = 0.2878976$, and we enter that into the pwr.anova.test

## 
##      Balanced one-way analysis of variance power calculation 
## 
##               k = 3
##               n = 39.76299
##               f = 0.2878976
##       sig.level = 0.05
##           power = 0.8
## 
## NOTE: n is number in each group

Here is where the LSD really shines. This standard test gives us an $n$ of 39.76299 per group for our three groups, giving an initial total of 119.28897. However, each group in the LSD is an individual result. For our experiment this is 9 results per day! So dividing our sample total by this number and rounding up, the total number of replicas is 14.

This means the experiment must run for exactly two weeks.

Run-time Logistics #

In our study, cart flow versions are assigned in real time. When a customer initiates a cart by clicking a buy action, we determine their city and audience segment on the spot. Audience segment classification follows a simple decision tree:

If the customer is logged in and previously identified as a Regular, they remain in that group.
If the customer is logged in and they are a Responder, they remain in that group.
For all other customers, we check whether they arrived via a coupon, affiliate or an email marketing link. If so, they are classified as a Responder in real-time.
Anyone not meeting the above conditions is assigned to the Other segment.

The city is identified from the logged in customer details or their IP address if they are not logged in.

From this information, the cart flow is assigned based on the LSD (no change for control group). We then track whether they complete the purchase, or abandon it. Timeouts are counted as abandonments.

Analysis #

As soon as the two weeks were up, we collected and analyzed the results. The raw data is a listing of all the started carts for each combination of city and audience, along with the version of the cart flow that they were exposed to according to the LSD and a 1 indicating cart completion or a 0 indicating abandonment.

A Quick Look at the Collected Data #

A short section of the summary by day, city and audience is shown below. We expect a number of rows equal to $14 \times 9:

`1`	`## COUNT (summary rows): 126`

`1`	`## HEAD (first 6 rows)`

##   day     city  audience    cart_flow dtc dcc       ccr
## 1   1  Hamburg     Other Easy Payment   8   3 0.3750000
## 2   1  Hamburg   Regular      Control  10   7 0.7000000
## 3   1  Hamburg Responder    Easy Edit   9   4 0.4444444
## 4   1 Istanbul     Other    Easy Edit   9   4 0.4444444
## 5   1 Istanbul   Regular Easy Payment   9   5 0.5555556
## 6   1 Istanbul Responder      Control   9   7 0.7777778

`1`	`## TAIL (last 6 rows)`

##     day     city  audience    cart_flow dtc dcc       ccr
## 121  14 Istanbul     Other    Easy Edit   7   5 0.7142857
## 122  14 Istanbul   Regular Easy Payment   8   4 0.5000000
## 123  14 Istanbul Responder      Control   7   4 0.5714286
## 124  14   Kraków     Other      Control   6   5 0.8333333
## 125  14   Kraków   Regular    Easy Edit   7   3 0.4285714
## 126  14   Kraków Responder Easy Payment   5   4 0.8000000

The ANOVA Table #

Now it’s time to run the ANOVA. Recall our model for this is:

$$ Y_{\text{ijk}} = \mu + \alpha_i + \beta_j + \tau_k + \epsilon_{\text{ijk}} $$

Plugging in city for $\alpha$ on rows, audience for $\beta$ on columns, along with our cart flow alternatives as the treatment factor $\tau$, take note that there should be no interaction terms with this design.

##              Df Sum Sq Mean Sq F value   Pr(>F)    
## cart_flow     2  0.263 0.13157   4.804  0.00985 ** 
## audience      2  0.194 0.09690   3.538  0.03215 *  
## city          2  0.550 0.27485  10.037 9.38e-05 ***
## Residuals   119  3.259 0.02738                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

From these results, we can see evidence against the null hypothesis that there is no difference in mean conversion rate between treatment groups. We’ll take a more detailed look at this outcome in the insights section.

Calculating Relative Efficiency #

The values in the ANOVA table also provide what we need to estimate the relative efficiency

`1`	`## DF residual: 119`

`1`	`## DF CRD: 123`

`1`	`## SSE: 3.258748`

`1`	`## SSE CRD (estimated): 4.00224`

`1`	`## RE: 1.188212`

Other Model Assumptions #

A visual check of the model assumptions does not reveal any problems.

Insights #

We can explore the magnitude of the effects from each shopping cart flow by creating a linear model from these data, beginning with the simplest model $\text{CCR} \sim \tau_k$, for the treatment factor alone:

## 
## Call:
## lm(formula = ccr ~ cart_flow, data = cart_day_summary)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.50383 -0.10348  0.00867  0.12951  0.34201 
## 
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)    
## (Intercept)            0.59247    0.02783  21.286  < 2e-16 ***
## cart_flowEasy Edit     0.11136    0.03936   2.829  0.00545 ** 
## cart_flowEasy Payment  0.06553    0.03936   1.665  0.09853 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1804 on 123 degrees of freedom
## Multiple R-squared:  0.06169,	Adjusted R-squared:  0.04643 
## F-statistic: 4.043 on 2 and 123 DF,  p-value: 0.01992

Here, we see the estimate for the Intercept (control group) of 0.5925 is very much what we expect given the prior month’s data. The Easy Payment cart flow of 0.0655 has just exceeded our MDE, while the Easy Edit cart flow of 0.1114 is slightly more than double.

Digging deeper to view the full model $\text{CCR} \sim \alpha_i + \beta_j + \tau_k$, we now see variance partitioned across the different groups in the LSD:

## 
## Call:
## lm(formula = ccr ~ city + audience + cart_flow, data = cart_day_summary)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.48091 -0.09506  0.01909  0.10904  0.38451 
## 
## Coefficients:
##                        Estimate Std. Error t value Pr(>|t|)    
## (Intercept)            0.549964   0.039005  14.100  < 2e-16 ***
## cityIstanbul          -0.062937   0.036111  -1.743  0.08394 .  
## cityKraków             0.097610   0.036111   2.703  0.00788 ** 
## audienceRegular        0.006557   0.036111   0.182  0.85622    
## audienceResponder      0.086279   0.036111   2.389  0.01845 *  
## cart_flowEasy Edit     0.111359   0.036111   3.084  0.00254 ** 
## cart_flowEasy Payment  0.065526   0.036111   1.815  0.07211 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1655 on 119 degrees of freedom
## Multiple R-squared:  0.236,	Adjusted R-squared:  0.1975 
## F-statistic: 6.127 on 6 and 119 DF,  p-value: 1.276e-05

For example the Intercept now represents just the customers in Hamburg in the Other audience segment. This group has a slightly lower CCR at 0.55 but our model shows us that we still expect the significant increase in CCR from the new versions of cart flow.

Conclusions #

We presented Latin Square Designs as an efficient way to run A/B tests while controlling for two known sources of variance. This method increases efficiency relative to a Complete Randomized Design. At the same time, it allows you to reduce the number of experimental samples while maintaining statistical power.

Our real-world example, an optimization for an e-commerce application, tested two potential improvements to the shopping cart flow, blocking over geographical and customer segments factors. It ran for two weeks and yielded a useful insight about which of the improvements to prioritize. Ultimately, both improvements could be deployed in quick succession, avoiding the opportunity cost of running separate, non-concurrent A/B tests.

The relative efficiency score of 1.1882 tells us that the variance was reduced by 15.84%, compared to the CRD. In addition, we also gained by the threefold decrease in the number of customers who needed to participate in the test, two thirds of whom saw a new version of the cart flow. This represents a huge reduction on the risk to these conversion opportunities.

Appendix #

Here you can find the source code.

library(knitr)
library(dplyr)
library(tidyr)
library(readr)
library(daewr)
library(agricolae)
library(pwr)

opts_chunk$set(warning = FALSE
              , message = FALSE
              , echo = FALSE
              )

hook_output <- knit_hooks$get("output")
knit_hooks$set(output = function(x, options) {
  lines <- options$output.lines
  if (is.null(lines)) {
    return(hook_output(x, options))  # pass to default hook
  }
  x <- unlist(strsplit(x, "\n"))
  more <- "..."
  if (length(lines)==1) {        # first n lines
    if (length(x) > lines) {
      # truncate the output, but add ....
      x <- c(head(x, lines), more)
    }
  } else {
    x <- c(more, x[lines], more)
  }
  # paste these lines together
  x <- paste(c(x, ""), collapse = "\n")
  hook_output(x, options)
})
rows <- c("I", "II", "III")
cols <- c("x", "y", "z")

latin_square <- matrix(c("A", "B", "C",
                         "B", "C", "A",
                         "C", "A", "B"),
                       nrow = 3, byrow = TRUE)

df <- as.data.frame(latin_square)
colnames(df) <- cols
rownames(df) <- rows

kable(df)
cart_flow <- c("Control", "Easy Payment", "Easy Edit")
outdesign = design.lsd(cart_flow, seed = 232)
lsd <- outdesign$book[, 2:4]
levels(lsd$row) <- c("Regular", "Responder", "Other")
levels(lsd$col) <- c("Hamburg", "Istanbul", "Kraków")
names(lsd) <- c("audience", "city", "cart_flow")

square_layout <- lsd %>% pivot_wider(
  names_from = audience,
  values_from = cart_flow
)
kable(square_layout)
average_days_per_month <- mean(
  c(rep(c(31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31), 3),
  c(31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31))
)

daily_started_carts <- c(55, 56, 62, 80, 57, 61, 61, 66, 59, 54, 44, 55, 61, 62, 59, 60, 63, 63, 58, 59, 72, 61, 50, 50, 55, 76, 60, 62, 55, 66, 60)
daily_completed_carts <- c(34, 31, 31, 45, 29, 41, 43, 37, 37, 30, 23, 40, 33, 39, 37, 36, 34, 47, 36, 35, 42, 36, 35, 33, 36, 44, 40, 42, 28, 50, 26)
daily_revenue <- c(1925.72, 689.1, 953.92, 1033, 1118.43, 1458, 1052.48, 520.65, 588.69, 439.11, 287.3, 1053.69, 859.18, 1276.8, 478.41, 685.41, 1124.76, 467.46, 828, 442.06, 850.5, 587.97, 542.43, 942.3, 1440.45, 2651.39, 2560.8, 692.23, 293.48, 686, 579.48)
average_daily_revenue <- mean(daily_revenue)

mde <- 0.055
expected_revenue_increase <- mde * average_daily_revenue * average_days_per_month

cart_completion_rate <- mean(daily_completed_carts / daily_started_carts)
cart_completion_rate_var <- var(daily_completed_carts / daily_started_carts)
target_cart_completion_rate <- cart_completion_rate + mde
factor_levels <- 3
p_vec <- c(cart_completion_rate, mean(c(cart_completion_rate, target_cart_completion_rate)), target_cart_completion_rate)
p_bar <- mean(p_vec)
ss_between <- sum((p_vec - p_bar)^2)
var_within <- cart_completion_rate_var
f <- sqrt(ss_between / (factor_levels * var_within))
res <- pwr.anova.test(k = 3, f = f, sig.level = 0.05, power = 0.8)
res
cart_data <- read_csv("cart_layout.csv")

cart_day_summary <- cart_data %>%
  group_by(day, city, audience, cart_flow) %>%
  summarise(
    dtc = n(),
    dcc = sum(completed),
    ccr = mean(completed),
    .groups = "drop"
  )

cat("COUNT (summary rows): ", nrow(cart_day_summary))

cat("HEAD (first 6 rows)")
head(as.data.frame(cart_day_summary))
cat("TAIL (last 6 rows)")
tail(as.data.frame(cart_day_summary))
mod_lsd <- aov(ccr ~ cart_flow + audience + city, data = cart_day_summary)
summary(mod_lsd)
df_lsd <- summary(mod_lsd)[[1]]["Residuals", "Df"]
cat("DF residual: ", df_lsd)
df_crd <- df_lsd + summary(mod_lsd)[[1]]["audience", "Df"] + summary(mod_lsd)[[1]]["city", "Df"]
cat("DF CRD: ", df_crd)
sse_lsd <- summary(mod_lsd)[[1]]["Residuals", "Sum Sq"]
cat("SSE: ", sse_lsd)
sse_crd <- sse_lsd + summary(mod_lsd)[[1]]["audience", "Sum Sq"] + summary(mod_lsd)[[1]]["city", "Sum Sq"]
cat("SSE CRD (estimated): ", sse_crd)
# Relative Efficiency
rel_eff <- (df_lsd * sse_crd) / (df_crd * sse_lsd)
cat("RE: ", rel_eff)
plot(mod_lsd, which = c(2, 5))
slm_tau <- summary(lm(ccr ~ cart_flow, data = cart_day_summary))
slm_tau
slm_abt <- summary(lm(ccr ~ city + audience + cart_flow, data = cart_day_summary))
slm_abt

To A/B or not to A/B

Sun, 16 Feb 2025 00:00:00 +0000

Introduction #

This is the story of a project that began as a straightforward A/B test but quickly revealed more than expected—offering fresh insights and expanding the scope of analysis.

It’s been a while since I worked as an independent data and analytics consultant. I went freelance after many years in data systems, BI, and MIS at a large multinational education company. During that time, I led projects using applied statistics, data mining, and algorithmic forecasting—and discovered a real passion for data science. But the chance to deepen those skills long-term wasn’t there, so I made the leap into freelancing, motivated by clarity about my goals and a desire for more hands-on, impactful work.

One of my first projects was with a startup language app—now a common genre, but at the time still rapidly evolving. These apps rely on gamified exercises, engaging features, and clever design to drive user retention and learning outcomes. Back then, much of that had to be figured out from scratch.

My client, still in the early stages, was looking to boost user growth with tactical feature launches tied to measurable impact. I was brought in to run a time-limited experiment: would a prototype for more interactive exercises lead to better word retention for new vocabulary? The results would inform whether to launch or revisit the design.

Disclaimer #

I’m grateful to that client for agreeing to partially lift our NDA so I could share this work. Enough time has passed—and the tech has evolved enough—that the study no longer holds commercial value. The company remains anonymous, and no personal or sensitive data is included here. The original dataset is not publicly available.

On the Product and Business Goals #

At the time of the project, the product used push notifications to prompt users to complete regular word retention exercises. These took the form of Flashcards, displaying a word in either the target or home (usually native) language. Users could tap buttons to flip the card, confirm memorisation, or skip. This was before gesture-based navigation was common on Android, so buttons were still the primary interface.

The new feature was more interactive—closer to a game. Users saw a sentence in the target language with a blank and had to choose the correct word to fill it from a set of options. The idea was that selecting the right word in context shows deeper understanding. Unlike Flashcards, which rely on self-reported memorisation, this approach generates more objective data on user progress. As a result, it could improve the accuracy of learning personalisation over time.

Strategically, the client was keen on Fill-in-the-Blank exercises for their potential to yield richer formative data. But they came with a trade-off: they were more cognitively demanding and time-consuming, raising concerns about lower engagement and increased churn.

Problem Definition #

The experiment compared two types of exercises—a simple two-level factor:

Flashcards (control group)
Fill-in-the-Blank (test group)

The primary goal was to assess whether these exercises led to different word retention scores, measured through separate weekly in-app vocabulary quizzes. We also aimed to evaluate differences in user engagement and churn between the two groups.

Experimental Hypotheses #

Word Retention #

$H_0$ There is no difference in word retention score that depends on the type of exercise - Flashcards vs. Fill-Blanks
$H_a$ There is a difference in word retention score that depends on the type of exercise - Flashcards vs. Fill-Blanks

Here, the hoped-for outcome is that $H_0$ can be rejected.

Churn Rate #

$H_0$ There is no difference in churn rate that depends on the type of exercise - Flashcards vs. Fill-Blanks
$H_a$ There is a difference in churn rate that depends on the type of exercise - Flashcards vs. Fill-Blanks

Here, the hoped-for outcome is that $H_0$ cannot be rejected.

Confounding Factors #

While designing the experiment, we quickly identified several potential confounding factors:

Engagement bias: If users engage differently with the two exercise types, retention scores could be affected independently of the exercise’s actual learning value. Since participation couldn’t be enforced, lower engagement with Fill-in-the-Blanks might reflect usability issues rather than learning efficacy. Some mitigation—like improving the prototype’s appeal—was considered.
Time-based effects: Learning outcomes or churn impacts might take longer than a single week to materialize. A short-term test could miss these delayed effects.
Survivor bias: If churn rates differ between groups, we risk measuring outcomes only among the more persistent users. This could inflate performance metrics for the group with higher dropout rates.

Hypothetical User Model #

To clarify some of our thinking about confounding factors, together with the client we developed a more sophisticated hypothesis about the existence of latent and causal factors. This is represented in the diagram below.

We developed a hypothetical model of the user, assuming each had a latent, unobserved ability. While not directly measurable, this ability could influence how engaged users were, how easily they retained vocabulary, and their likelihood of churn. Users with lower latent ability would likely face more difficulty and need greater resilience to stick with language learning.

As a startup focused on rapid growth, the client was especially wary of anything that might increase churn. Even a short-term experiment carrying that risk was a concern. Minimising this risk—while still running a meaningful test—was a key part of my role, and heavily influenced both the experimental design and the analytical approach.

Experimental Design #

Given the confounding factors, a simple A/B test followed by a t-test at the end of each week wouldn’t suffice. That approach would overlook two key sources of bias:

Indirect effects: Fill-in-the-Blank exercises could affect retention scores indirectly by altering engagement levels.
Survivor bias: If lower-performing users churned at higher rates due to the added effort required, this would skew group averages.

To address this, I proposed a set of guiding principles to structure the experiment in a way that would let us assess these effects in a single pass:

Random assignment: To control for latent ability and user preferences, users were randomly assigned to one exercise type.
Consistent exposure: Each group remained on the same activity for the full duration to allow us to detect differences in engagement and churn over time.
Limited duration: The experiment would run for four weeks to reduce the risk of users churning simply due to prolonged exposure to a potentially less enjoyable feature.
Minimised exposure: We limited the number of users in the study to further reduce potential churn impact during this critical growth phase.

Sample Size Calculation #

The client agreed on a minimum detectable effect (MDE) of 2 additional words retained per week. This would translate to a gain of roughly 108 words annually—modest, but meaningful—on top of a baseline of about 20 words per week (or 1,040 per year). Naturally, they were hoping for more, but this was set as the threshold for meaningful improvement.

Historical data showed a standard deviation of 4.87 in weekly word retention scores, which provided a useful input for estimating sample size. However, we also had to factor in churn: around 25% of users didn’t engage with the app for at least one week during any four-week period. To ensure we had sufficient completions to maintain 80% statistical power, I adjusted for this dropout risk.

I used the short-hand formula -

$$N \approx \left(\frac{8\sigma}{\Delta}\right)^2$$ to estimate the total sample size, which gave me

$$ \left\lceil \left(\frac{8 \times 4.87}{2}\right)^2 \right\rceil = 380 $$ This gave us a quick, practical estimate to review feasibility with the client.

Later, I validated the result using power.t.test in R and adjusted for churn by inflating the required sample size, using a conservative adjustment based on the churn rate plus two standard deviations (via $\sigma^2 = p(1-p)$).

mde <- 2
pilot_sd_retention <- 4.87
power <- 0.8          # Desired power (80%)
alpha <- 0.05         # Significance level
four_week_churn_rate <- 0.25
sd_four_week_churn_rate <- sqrt(four_week_churn_rate * (1 - four_week_churn_rate))

# calculates the required sample size per group
sample_size <- pwr.t.test(
  d = mde / pilot_sd_retention,
  power = power,
  sig.level = alpha,
  type = "two.sample"
)$n

adjusted_sample_size = ceiling((1 + four_week_churn_rate + 2 * sd_four_week_churn_rate) * sample_size)

The final result was:

`1`	`## Recommended sample size per group: 200`

I don’t know why but it was weirdly satisfying that such a nice round number popped out by pure chance.

Participation in the Trial and Managing Churn #

Although the sample size was padded to account for churn, we still needed to monitor it closely during the trial. Crucially, we couldn’t exclude users who dropped out—doing so would introduce survivor bias. Users had to be entered into the trial at random, and their outcomes included, regardless of whether they completed the full period.

From a larger pool of about 10,000 users, we randomly selected 400 users using R’s random number generator and assigned them to one of two groups. For exactly four weeks, each participant received two push notifications per week, prompting them to complete a vocabulary retention exercise based on words previously encountered in the app.

Each user was assigned a set of metrics:

Per-user:

exercise_type (treatment factor): “Fill-Blanks” or “Flashcards” — the only exercise type they saw during the trial.
churn_week (int): The first week in which the user didn’t use the app at all; if they were active all four weeks, this was set to 5.
churn (factor): “churned” if churn_week < 5; otherwise, “completed trial”.

Per-user, per-week:

engagement (factor): “High” if the user completed at least one exercise that week, “Low” otherwise. If a user churned, engagement was set to “Low” for that week and all subsequent ones.
retention (int): The number of words correctly retained in the weekly quiz. If the user churned, this was set to NA for that and all future weeks.

Interlude #

In the meantime while I was agreeing these parameters with the client and his team, the developers were preparing the A/B test framework. They confirmed that once a user was allocated to a test group, they would only see one type of test for the four week duration and the requested data would be collected weekly. All other users would continue with business as usual.

There was nothing left to do but run the trial. I took a short SCUBA diving break and then worked with another client for a couple of weeks.

Results Analysis Part 1 - Preliminaries #

Immediately after the test period completed, I collated the data for an initial review.

##    engagement       retention    
##  Min.   :0.0000   Min.   : 7.00  
##  1st Qu.:0.0000   1st Qu.:19.00  
##  Median :1.0000   Median :22.00  
##  Mean   :0.7344   Mean   :22.19  
##  3rd Qu.:1.0000   3rd Qu.:25.00  
##  Max.   :1.0000   Max.   :38.00  
##                   NA's   :223

Here we can see that the grand mean engagement over all users, all weeks was 0.73438, while the mean word retention each week was 22.19317. The number of NAs in this result set looks high but there are duplicates. A user who churned in week 1 would present as 4 NAs. The actual number of churned users was counted separately but can also be determined by looking at the fourth week results only. This was found to be 90, which is 0.22 on a total N of 400 (two groups) and slightly below our pre-study expectation.

Summary Statistics by Treatment/Control and Weeks #

I tabulated all the weekly metrics that we had collected and summarised their means and standard deviations, as shown here.

## # A tibble: 8 × 6
##   exercise_type  week retention retention_sd engagament engagament_sd
##   <fct>         <int>     <dbl>        <dbl>      <dbl>         <dbl>
## 1 Flashcards        1      21.2         4.62      0.83          0.377
## 2 Flashcards        2      21.7         4.09      0.875         0.332
## 3 Flashcards        3      21.3         4.17      0.835         0.372
## 4 Flashcards        4      22.1         4.14      0.79          0.408
## 5 Fill-Blanks       1      21.8         4.99      0.655         0.477
## 6 Fill-Blanks       2      22.7         5.06      0.64          0.481
## 7 Fill-Blanks       3      22.9         5.22      0.615         0.488
## 8 Fill-Blanks       4      24.2         4.91      0.635         0.483

Observed Four Week Churn Rates #

Aside from the retention and engagement means, we also took the opportunity to check the churn rate. In particular, the four week churn rate was of interest.

## # A tibble: 2 × 3
##   exercise_type  week churn_rate
##   <fct>         <int>      <dbl>
## 1 Flashcards        4      0.205
## 2 Fill-Blanks       4      0.245

The rate for Fill-Blanks was in line with the prior expectations but the rate for Flashcards was a bit lower. Was this significant? I did a quick log odds ratio test to find out.

## 
## z test of coefficients:
## 
##                                                Estimate Std. Error z value
## completed trial:churned/Fill-Blanks:Flashcards -0.22987    0.24023 -0.9569
##                                                Pr(>|z|)
## completed trial:churned/Fill-Blanks:Flashcards   0.3386

The large p-value indicated that that their was no evidence of a significant difference. I plotted the counts to double check.

The non-finding was backed up by the fourfold plot, which showed overlapping confidence intervals of the quarters. The difference in churn rates was entirely within the margin of error. Nevertheless, I chose to reserve my final judgement until I had a chance to look more deeply at the other results.

Naive T-Test #

Sometimes, the only required analysis for an A/B test is a t-test to check for a statistical difference in means between groups, assuming everything has gone well with the experiment. We knew that wasn’t the case here but it’s always an informative measure.

I started by taking mean weekly retention score per user, so long as they didn’t churn in the first week of the study. This was calculated by summing up their individual weekly scores and dividing by the number of weeks that they remained unchurned.

`1`	`## Non-churned grand mean retention: 22.17154`

Then I ran the Student’s t-test to get an initial intuition of whether the whole endeavor had yielded a statistically interesting result.

## 
## 	Welch Two Sample t-test
## 
## data:  fill_blanks and flash_cards
## t = 4.5537, df = 360.61, p-value = 7.219e-06
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.7751163 1.9534842
## sample estimates:
## mean of Fill-Blanks  mean of Flashcards 
##            22.86821            21.50391

These results suggested a statistically significant difference in means of between 0.77512 and 1.95348 with a confidence of 0.95 among users who were given Fill-Blanks exercises over users who were given Flashcards exercises. I noted that the upper bound was below the minimum detectable effect of 2 that we had agreed before starting but it was too early to be disappointed. The possible presence of confounding factors required a deeper analysis.

Results Analysis Part 2 - Engagement #

One of my first priorities was to verify some of our initial assumptions. Did the level of engagement depend on the exercise itself? Also, over the weeks of the study, did the users show a growing or diminishing level of engagement? This information would shine a light on our hypothetical user model and reveal any important interactions.

Analysis of Variance #

The quickest way to check is with an ANOVA test.

##                      Df Sum Sq Mean Sq F value Pr(>F)    
## week                  1   0.30   0.300   1.616  0.204    
## exercise_type         1  15.41  15.406  82.960 <2e-16 ***
## week:exercise_type    1   0.03   0.028   0.151  0.697    
## Residuals          1596 296.38   0.186                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

According to this result, there is a statistically significant effect on engagement depending on the type of exercise, but there is no detectable difference each week.

Mosaic Visualisation and Significance Tests #

A visual analysis can help to better understand what is happening.

I’m a strong advocate for mosaic plots when working with count data like this. I’ve taught workshops on them because, while they can be unfamiliar at first, they’re incredibly revealing once you know how to read them.

The plot recursively divides a canvas by the specified categorical factors, with tile area proportional to counts at each intersection. If the factors are independent, the result is a neat grid. The more skewed or uneven the tiles, the stronger the interactions between variables. Tiles are shaded by Pearson residuals from a $\chi^2$ test of independence — essentially embedding a statistical test into the visualization. Blue tiles (positive residuals) mark over-represented combinations; red tiles (negative residuals), under-represented ones.

What did this analysis reveal about user engagement?

As expected, the distribution across exercise type (Flashcards vs. Fill-Blanks) and week was even, since users were assigned randomly and engagement was recorded weekly regardless of churn. The striking difference appeared in engagement levels: Low engagement was clearly over-represented among Fill-Blanks users.

The implication was straightforward — users were more likely to complete Flashcard exercises. This aligned with expectations: Flashcards are quicker and cognitively lighter — users can simply tap through and self-confirm memorisation. The pattern held consistently over time, suggesting the effect was inherent to the mechanics of the exercises rather than a temporary novelty or fatigue effect.

Causal Analysis #

The key question was this: How much did lower engagement with Fill-in-the-Blank exercises affect overall word retention?

The client and I had anticipated that these exercises might see lower engagement — or even higher churn — which would be counterproductive, especially if it masked what could otherwise be better learning outcomes. To understand this dynamic, I needed to go beyond basic group comparisons and look into causal pathways.

I was already somewhat familiar with structural equation modelling (SEM) from past work on student surveys, but this problem—where an independent variable (exercise type) may influence the response (retention) indirectly through a mediator (engagement)—called for a more targeted approach.

While researching, I came across the mediation package in R, which seemed tailor-made for this use case. To build confidence in the results, I ran two parallel analyses: one using the mediation package, and another using a more traditional SEM approach, allowing me to compare the outputs and validate the findings.

Mediation Analysis #

## 
## Causal Mediation Analysis 
## 
## Nonparametric Bootstrap Confidence Intervals with the Percentile Method
## 
##                Estimate 95% CI Lower 95% CI Upper p-value    
## ACME             -0.219       -0.343        -0.09  <2e-16 ***
## ADE               1.513        1.023         2.01  <2e-16 ***
## Total Effect      1.294        0.808         1.77  <2e-16 ***
## Prop. Mediated   -0.169       -0.304        -0.06  <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Sample Size Used: 1377 
## 
## 
## Simulations: 10

This model does not take into account the repeated measures occuring each week. Despite the earlier ANOVA test and mosaic plot confirming that this wasn’t a particular concern here, I wanted to be very cautious when interpreting the results numerically.

The summary above shows a significant result for Average Causal Mediation Effect (ACME) of -0.21886. Any significant result here is evidence to reject a null hypothesis of no indirect effect. The ACME has the opposite sign of the Average Direct Effect (ADE) at 1.51281, which suggests that the total effect is less than it otherwise would have been.

I checked alignment with my preliminary result (from the earlier t-test) and found that the total effect of 1.29395 is close to the mean difference between groups of -1.3643. So this causal analysis seems strongly to suggest that the mean difference would have been larger by 0.21886 or some quantity between the ACME confidence interval, shown in the summary above.

Structural Equation Model #

I have used SEM to model latent factors before but had not tried using them to model a mediation effect. Anyway, the approach is pretty much the same except that there aren’t any latent factors, only manifest items. I conducted the analysis in much the same way as I would usually do it.

## lavaan 0.6-19 ended normally after 1 iteration
## 
##   Estimator                                         ML
##   Optimization method                           NLMINB
##   Number of model parameters                         5
## 
##   Number of observations                          1377
## 
## Model Test User Model:
##                                                       
##   Test statistic                                 0.000
##   Degrees of freedom                                 0
## 
## Parameter Estimates:
## 
##   Standard errors                             Standard
##   Information                                 Expected
##   Information saturated (h1) model          Structured
## 
## Regressions:
##                    Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
##   engagement ~                                                          
##     exercise_type    -0.187    0.018  -10.179    0.000   -0.187   -0.265
##   retention ~                                                           
##     engagement        1.169    0.369    3.164    0.002    1.169    0.087
##     exercise_type     1.513    0.262    5.785    0.000    1.513    0.160
## 
## Variances:
##                    Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
##    .engagement        0.116    0.004   26.239    0.000    0.116    0.930
##    .retention        21.885    0.834   26.239    0.000   21.885    0.974

Here we’re seeing very well aligned results to the mediation method, with a direct effect of exercise_type on retention of 1.51281 and an indirect effect of exercise_type on engagement of -0.18724.

It’s also possible to plot the SEM, which is why I wanted to run the analysis this way.

This was a very nice, simple visual for explaining to the client that Fill-Blanks has a downward effect on engagement, which must have a dampening effect on any increases in retention for users in the Fill-Blanks group.

Results Analysis Part 3 - Word Retention #

This stage of the analysis required the use of a linear mixed model (LMM) for one simple reason; given that we were tracking the users over the course of four weeks, with one overall retention measure per week, the experimental design is repeated samples, which violates the independence assumptions of an OLS linear model. I proceeded by working through a hierarchy of models, increasing the number interaction terms, to determine the best fitting model.

Linear Mixed Model Selection and Analysis #

As LMM are quite a bit more complex than linear models, there is a lot more to the summaries and consequently more console output. For brevity, I only post the best fitting model summary here, after the ANOVA test to determine the best fit.

## Data: selected_users
## Models:
## mm1: retention ~ (1 | user_id) + engagement + exercise_type + week
## mm2: retention ~ (1 | user_id) + engagement + exercise_type * week
## mm3: retention ~ (1 | user_id) + engagement * exercise_type * week
##     npar    AIC    BIC  logLik deviance  Chisq Df Pr(>Chisq)  
## mm1    6 8134.6 8166.0 -4061.3   8122.6                       
## mm2    7 8130.4 8167.0 -4058.2   8116.4 6.2290  1    0.01257 *
## mm3   10 8134.4 8186.7 -4057.2   8114.4 1.9689  3    0.57889  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Above you can see the model definitions. The notation shows that user_id has been set as a grouping variable with different intercepts but not different slopes. This means that the model’s intercept is modulated by each user’s random effect (their latent ability, in our user model).

Model mm2, with an interaction term between exercise_type and week is found to be a significantly better fit to the data than model one. The additional interaction term in model mm3 does nothing to improve on mm2. Therefore, mm2 was selected as the best fitting model. The model summary and diagnostics are shown next.

## Linear mixed model fit by maximum likelihood . t-tests use Satterthwaite's
##   method [lmerModLmerTest]
## Formula: retention ~ (1 | user_id) + engagement + exercise_type * week
##    Data: selected_users
## 
##      AIC      BIC   logLik deviance df.resid 
##   8130.4   8167.0  -4058.2   8116.4     1370 
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.3654 -0.6649 -0.0156  0.6648  3.1411 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  user_id  (Intercept)  2.401   1.55    
##  Residual             19.184   4.38    
## Number of obs: 1377, groups:  user_id, 376
## 
## Fixed effects:
##                                Estimate Std. Error        df t value Pr(>|t|)
## (Intercept)                     20.3160     0.5138 1367.1255  39.537   <2e-16
## engagement                       0.8852     0.3658 1356.5889   2.420   0.0157
## exercise_typeFill-Blanks         0.1858     0.5914 1370.9265   0.314   0.7535
## week                             0.1671     0.1497 1066.8353   1.116   0.2645
## exercise_typeFill-Blanks:week    0.5330     0.2132 1063.9878   2.500   0.0126
##                                  
## (Intercept)                   ***
## engagement                    *  
## exercise_typeFill-Blanks         
## week                             
## exercise_typeFill-Blanks:week *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) enggmn ex_F-B week  
## engagement  -0.601                     
## exrcs_tyF-B -0.622  0.111              
## week        -0.634 -0.101  0.592       
## exrcs_tF-B:  0.484  0.006 -0.867 -0.696

Note that there are 376 groups (user_id) discovered, which accounts for all users who didn’t churn in the first week. Then there are 1377 observations overall, which accounts for NAs for users who churned on any week after that, providing at least one but fewer than four observations to the study.

Model Diagnostics #

The model converged without any problems and the scaled residuals are centred on zero. This is also visible in the residual plots. The qqplot looked perfectly normal (no pun intended), and the random effects plot showed symmetry around zero, suggesting a normal distribution for the latent ability factor, as one would expect.

`1`	`## $user_id`

Sense-Making the Linear Predictor for Word Retention #

The model coefficients, as given in the summary, are significant for engagement, week, and the interaction between exercise_type * week but notably not for exercise_type main effect. This indicated to me that there is a significant difference between the two exercises but it does not become apparent until the later weeks of the study. I simplified the model accordingly by removing the non-significant main effect and used the ANOVA test to ensure that my simpler model was just as effective in explaining the variance in the retention scores.

## Data: selected_users
## Models:
## mma: retention ~ (1 | user_id) + engagement + exercise_type:week
## mm2: retention ~ (1 | user_id) + engagement + exercise_type * week
##     npar    AIC    BIC  logLik deviance  Chisq Df Pr(>Chisq)
## mma    6 8128.5 8159.9 -4058.3   8116.5                     
## mm2    7 8130.4 8167.0 -4058.2   8116.4 0.0986  1     0.7535

The Simplified Model and Easy Explanations #

The parameter set of the simplified model was much easier to explain to the client.

##                                 Estimate     Pr(>|t|)
## (Intercept)                   20.4163911 < 2e-16 *** 
## engagement                     0.8723426 0.0165 *    
## exercise_typeFlashcards:week   0.1393055 0.2486      
## exercise_typeFill-Blanks:week  0.7302830 1.19e-09 ***

The interpretation of these results is as follows:

the baseline expectation is 20.41639 but has some variability, given individual users’ innate ability
two standard deviations of this user variability on the baseline is plus or minus 1.96 * 1.55007 (possibly predictable in future from data collected from in app activities)
if engagement is High, there is an additional 0.87234
for each week up to four (so as not to extrapolate beyond the experimental conditions), there is an additional 0.73028 for Fill-Blanks exercises
the 0.13931 per week for Flashcards exercises is within the margin of error and can be ignored
as a result, there is a difference between the word retention scores for Fill-Blanks vs. Flashcards that accumulates over four weeks to 2.92112 (or more meaningfully, 3 additional words) compared to using only Flashcards
additionally to keep in mind, the more time-and-effort consuming Fill-Blanks tasks is depressing the engagement level, which is counter-productive to the average retention scores for those same users

This plain English explanation made the most sense to my client and their team. I did not attempt to fully quantify the last point because it was the output of a separate modelling procedure. I did explain, however, that we should really be seeing a different engagement coefficient per exercise type but the LMM was not a suitable tool for discovering it because it does not discover relationships between independent variables. Rather, these relationships introduce bias.

A Mildly Shocking Oversight #

Going back to the experimental design parameters, I suddenly realised that our sample size was flawed. This difference in retention scores over four weeks is statistically significant to at least a 95% confidence level. The MDE was set at 2, which meant that I could be sure with 80% confidence that this difference was not in danger of being a false positive,

Unfortunately, I had made an error with the sample size calculation. The client had sought confirmation (with power 0.8) of their MDE of 2 words per week for a total of 8! This would have resulted in a drastically smaller sample size. All things considered, this issue was not a material cause for concern in the end because a much smaller sample could easily have been thrown off by an unusually high churn week, for example. Also, the numbers were still very small with respect to the total number of users. I can laugh about it now but I was quite embarrassed at the time.

Results Analysis Part 4 - Churn Rate #

I was still tasked with identifying whether the extra effort of completing a Fill-Blanks exercise would have an adverse effect on churn rate. So far, I had a bit of evidence from the four week churn rate for just the Fill-Blanks group staying in line with prior expectations. Also, even though the Flashcards control group did appear to be a little lower, there was the non-significant log odds ratio test, reducing any major concern. Nevertheless, I wanted to complete the analysis that I had planned and that data had been collected for.

I proceeded with a non-parametric survival analysis by using the per user data described above. This contains the churn week or 5 if they made it to the end of the study (as the majority did). A boolean churn event of TRUE for those who did churn, and a weekly average retention for the weeks they participated. Finally, there was also the exercise type, of course.

Between Groups Log-Rank Test #

It’s easy to compare survival distributions between groups using the log-rank test

## Call:
## survdiff(formula = surv_obj ~ exercise_type, data = selected_churn)
## 
##                             N Observed Expected (O-E)^2/E (O-E)^2/V
## exercise_type=Flashcards  200       41     45.8     0.497      1.08
## exercise_type=Fill-Blanks 200       49     44.2     0.514      1.08
## 
##  Chisq= 1.1  on 1 degrees of freedom, p= 0.3

The test shows that there is no significant difference between the two groups’ survival. However, this method does not control for retention, which is a concrete measure of the user’s learning progress. Our hypothetical user model, based on intuition and experience in the sector, suggested that it was a possibility that users who felt they were under-performing might be more likely to churn. So I went ahead and checked that as well.

Cox’s Proportional Hazards Model With A Continuous Variable #

I’ve discussed Proportional Hazard (PH) models in a lot more detail in this post so I will skip over the intricacies and just say that if we find a high value for hazard at any given moment, the survival rate is falling fast. Conversely, we might say that the churn rate is going up. It’s usual to report on the log hazard and check for values that a greater than zero with statistical significance.

For continuous independent variables, it is necessary to fit a smooth spline predictor in place of the raw data. The model reports on the linear part and non-linear part separately.

## Call:
## coxph(formula = surv_obj ~ exercise_type + pspline(retention, 
##     df = 2), data = selected_churn)
## 
##                              coef se(coef)     se2   Chisq   DF       p
## exercise_typeFill-Blanks   0.0366   0.2561  0.2559  0.0205 1.00    0.89
## pspline(retention, df = 2 -0.0490   0.0362  0.0362  1.8377 1.00    0.18
## pspline(retention, df = 2                          16.6278 1.05 5.1e-05
## 
## Iterations: 3 outer, 11 Newton-Raphson
##      Theta= 0.813 
## Degrees of freedom for terms= 1.0 2.1 
## Likelihood ratio test=21.1  on 3.05 df, p=1e-04
## n= 376, number of events= 66 
##    (24 observations deleted due to missingness)

Notice here the number of observations used in the model is 376, just as we saw with the LMM. Given that there were 400 individuals to begin with, this means 24 individuals churned on week one and contributed no retention data. This made a full analysis impossible but I noticed that only the non-linear partition of the retention data returned a significant result.

The PH terms can be plotted, which sometimes helps with a non-intuitive analysis as we have here.

We see from these plots that the partial PH for the exercise types is close to zero, as the modeling suggests. The retention curve does appear to show a significant divergence from zero, certainly at the lower end. This aligned with the ideas we had in creating the user model at the beginning of the project - the mechanism being that users who are doing less well at memorising vocabulary are more likely to get frustrated stop returning to the app for many days at a time, or for good.

Survival Model of Churn #

As a final sense check, I created a survival model. Think of it as the reverse of churn. The greater the survival rate, the lower the churn rate in a direct $p, 1 - p$ relationship. The survival plot provided me with the ideal way to communicate my churn rate analysis to the client because it is such an easy to understand visual.

Certainly there is the appearance of a slightly lower survival rate for the Fill-Blanks group but it is very small and well within the margin of error. Happily, we could pretty much rule out the new Fill-Blanks exercise as a significant cause of increased churn risk.

Summary #

What began as a simple A/B test for a new vocabulary exercise evolved into a more complex analysis once we examined the client’s goals and the dynamics of the product. A basic t-test wouldn’t have captured the nuanced interactions at play.

By developing a user model with latent ability as a key concept, we identified several important confounding and mediating variables—particularly engagement and churn—that needed to be accounted for. This led to a multi-variate analysis approach, including a linear mixed model to handle repeated measures per user and individual-level variance.

To explore the causal pathway between exercise type, engagement, and retention, I used two methods: the mediation package and structural equation modelling (SEM). While I didn’t quantify the indirect effect precisely, both analyses helped clarify that engagement did mediate the impact of the exercise format on retention — albeit to a modest degree.

Key findings:

The new Fill-Blanks exercises improved average word retention by approximately 3 words over four weeks.
This improvement was lower than the client had hoped.
The indirect negative effect of reduced engagement was present but relatively small in magnitude.
Most importantly, we found no evidence of increased churn risk among users exposed only to Fill-Blanks exercises.

Conclusions #

This project was a powerful reminder that data science is about more than running models — it’s about framing the right questions and understanding the context in which data is generated.

The client’s openness to a deeper analysis allowed us to uncover insights that went beyond surface-level metrics. Although the headline result wasn’t transformative in terms of retention gains, the finding that Fill-Blanks exercises didn’t drive up churn gave the client confidence to move forward with the launch. The feature offered strategic advantages: better formative assessment data and continued product evolution to keep users engaged.

For me, this work reinforced a key lesson: deep domain understanding, clear causal thinking, and honest evaluation of trade-offs are what transform a data project from informative to actionable. It was a rewarding collaboration—and a great example of the kind of analytical thinking I always aim to bring to client work.

Appendix #

Here you can find the source code.

library(knitr)
library(tidyr)
library(dplyr)
library(survival)
library(survminer)
library(pwr)
library(vcd)
library(lme4)
library(lmerTest)
library(ggdag)
library(mediation)
library(lavaan)
library(semPlot)

opts_chunk$set(warning = FALSE
              , message = FALSE
              , echo = FALSE
              )

hook_output <- knit_hooks$get("output")
knit_hooks$set(output = function(x, options) {
  lines <- options$output.lines
  if (is.null(lines)) {
    return(hook_output(x, options))  # pass to default hook
  }
  x <- unlist(strsplit(x, "\n"))
  more <- "..."
  if (length(lines)==1) {        # first n lines
    if (length(x) > lines) {
      # truncate the output, but add ....
      x <- c(head(x, lines), more)
    }
  } else {
    x <- c(more, x[lines], more)
  }
  # paste these lines together
  x <- paste(c(x, ""), collapse = "\n")
  hook_output(x, options)
})

par(mar = c(4,3,3,1))
source("HeartTheme.R")
load("retention.RData")
selected_users <- as_tibble(selected_users) %>%
  mutate(exercise_type = relevel(exercise_type, "Flashcards")) # set the control group
dag <- dagify(
  Engagement ~ Ability + "Exercise Type",
  Retention ~ Engagement + Ability + "Exercise Type",
  Churn ~ Retention + Engagement + Ability,
  latent = "Ability",
  exposure = "Exercise Type",
  outcome = "Churn"
)

tidy_dag <- tidy_dagitty(dag, seed = 222, layout = "nicely")

ggdag(tidy_dag, text = FALSE, use_labels = "name") + theme_dag()
mde <- 2
pilot_sd_retention <- 4.87
power <- 0.8          # Desired power (80%)
alpha <- 0.05         # Significance level
four_week_churn_rate <- 0.25
sd_four_week_churn_rate <- sqrt(four_week_churn_rate * (1 - four_week_churn_rate))

# calculates the required sample size per group
sample_size <- pwr.t.test(
  d = mde / pilot_sd_retention,
  power = power,
  sig.level = alpha,
  type = "two.sample"
)$n

adjusted_sample_size = ceiling((1 + four_week_churn_rate + 2 * sd_four_week_churn_rate) * sample_size)
cat("Recommended sample size per group: ", adjusted_sample_size)
summary(dplyr::select(selected_users, engagement, retention))
mean_engagement <- round(mean(selected_users$engagement), 5)
mean_retention <- round(mean(selected_users$retention, na.rm = TRUE), 5)
na_retention <- sum(is.na(selected_users$retention))
na_fourth_week <- sum(is.na(matrix(selected_users$retention, 2 * adjusted_sample_size, 4)[, 4]))
selected_users %>% 
  rename(ret = retention) %>%
  group_by(exercise_type, week) %>%
  summarise(
    retention = mean(ret, na.rm = TRUE), retention_sd = sd(ret, na.rm = TRUE),
    engagament = mean(engagement, na.rm = TRUE), engagament_sd = sd(engagement, na.rm = TRUE),
    .groups = "drop"
  )
selected_users %>% 
  rename(ret = retention) %>%
  group_by(exercise_type, week) %>%
  summarise(
    churn_rate = mean(is.na(ret)),
    .groups = "drop"
  ) %>%
  dplyr::filter(week == 4)
churn_ratio <- selected_churn %>%
      dplyr::select(exercise_type, churn) %>%
      mutate(churn = factor(churn, labels = c("completed trial", "churned")))

lr <- loddsratio(churn ~ exercise_type, data = churn_ratio)
summary(lr)
fourfold(table(churn_ratio))
cat("Non-churned grand mean retention: ", round(mean(selected_churn$retention, na.rm = TRUE), 5))
fill_blanks <- selected_churn[selected_churn$exercise_type == "Fill-Blanks", "retention"]
flash_cards <- selected_churn[selected_churn$exercise_type == "Flashcards", "retention"]
tt <- t.test(fill_blanks, flash_cards, na.action = omit.rm, var.equal = FALSE) 

names(tt$estimate) <- paste("mean of", c("Fill-Blanks", "Flashcards"))
tt
summary(aov(engagement ~ week * exercise_type, data = selected_users, na.action = na.omit))
mosaic(
  table(
  selected_users %>%
    dplyr::mutate(engagement = factor(engagement, labels = c("Low", "High"))) %>%
    dplyr::select(exercise_type, week, engagement)
  ),
  shade = TRUE
)
# tidy up the data 
selected_users_dropna <- selected_users %>%
  filter(!is.na(retention))

set.seed(135)
med_model <- lm(engagement ~ exercise_type, data = selected_users_dropna)
outcome_model <- lm(retention ~ exercise_type + engagement, data = selected_users_dropna)
mediation_result <- mediate(med_model, outcome_model, 
                            treat = "exercise_type", mediator = "engagement", 
                            boot = TRUE, sims = 10)

summary_med <- summary(mediation_result)
summary_med
set.seed(135)
sem_model <- '
  engagement ~ exercise_type
  retention ~ engagement + exercise_type
'

# Fit SEM Model
fit <- sem(sem_model, data = selected_users_dropna)
summary_sem <- summary(fit, standardized = TRUE)
summary_sem
semPaths(fit, what="est", fade=FALSE, residuals=FALSE, edge.label.cex=0.75)
mm1 <- lmer(retention ~ (1 | user_id) + engagement + exercise_type + week, data = selected_users, REML = FALSE)
mm2 <- lmer(retention ~ (1 | user_id) + engagement + exercise_type * week, data = selected_users, REML = FALSE)
mm3 <- lmer(retention ~ (1 | user_id) + engagement * exercise_type * week, data = selected_users, REML = FALSE)
anova(mm1, mm2, mm3)
summary(mm2)
plot(mm2)
qqmath(resid(mm2))
dotplot(ranef(mm2, whichel = "user_id"), main = FALSE, scales = list(y = list(draw = FALSE)))
mma <- lmer(retention ~ (1 | user_id) + engagement + exercise_type:week, data = selected_users, REML = FALSE)
anova(mma, mm2)
coef_summary <- as.data.frame(coef(summary(mma))[,c("Estimate", "Pr(>|t|)")])
p_vals <- coef_summary[, 2]
stars <- ifelse(
  p_vals > 0.05,
  "",
  ifelse(
    p_vals > 0.01,
    " *",
    " ***"
  )
)
p_vals <- ifelse(
  p_vals < 2e-16,
  "< 2e-16",
  ifelse(
    p_vals < 0.01,
    formatC(p_vals, format = "e", digits = 2),
    as.character(round(p_vals, 4))
  )
)
p_vals <- format(paste0(p_vals, stars), justify = "left")
coef_summary$`Pr(>|t|)` <- p_vals


mma_intercept <- round(coef_summary["(Intercept)", "Estimate"], 5)
mma_intercept_sd <- round(sqrt(VarCorr(mma)[[1]][1]), 5)
mma_engagement <- round(coef_summary["engagement", "Estimate"], 5)
mma_fillblanks_week <- round(coef_summary["exercise_typeFill-Blanks:week", "Estimate"], 5)
mma_flashcards_week <- round(coef_summary["exercise_typeFlashcards:week", "Estimate"], 5)

coef_summary
selected_churn$exercise_type <- relevel(selected_churn$exercise_type, "Flashcards")
surv_obj <- Surv(time = selected_churn$churn_week, event = selected_churn$churn)
survdiff(formula = surv_obj ~ exercise_type, data = selected_churn)
cox_model <- coxph(formula = surv_obj ~ exercise_type + pspline(retention, df = 2), data = selected_churn)
cox_model
termplot(cox_model, se = TRUE, terms = 1, ylabs = "Log hazard", xlabs = "Exercise Type")
abline(h = 0, col="grey", lty=3)

termplot(cox_model, se = TRUE, terms = 2, ylabs = "Log hazard", xlabs = "Retention")
abline(h = 0, col="grey", lty=3)
surv_fit <- survfit(surv_obj ~ exercise_type, data = selected_churn)
surv_fit <- survfit(surv_obj ~ exercise_type, data = selected_churn)
ggsurvplot(surv_fit, data = selected_churn, pval = TRUE, conf.int = TRUE,
           legend.labs = c("Flashcards", "Fill-Blanks"),
           xlab = "Weeks", ylab = "Survival Probability",
           ylim = c(0.7, 1.0))

Tallinn Ride Hailing App

Mon, 27 Jan 2025 00:00:00 +0000

Introduction #

I acquired a fun little data set for a well-known ride-hailing app recently. I performed a pretty detailed analysis at the request of my source, including some clustering of the ride start locations. The idea was to help drivers plan ahead to get into position before times of peak demand. There’s no NDA and this data is no longer very fresh, so I thought it would be nice to show the results in an interactive Tableau viz.

Mapping With Tableau #

Mapping with Tableau is an absolute breeze. Just set the Latitude and Longitude points as discrete or dimension and be sure that the geographical role is set. Tableau usually figures this out unless the column names are not obviously inferred as Lat and Long.

For this viz, because there are hundreds of thousands of data points, I had to set up the clustered and aggregated sheets first and add the filters. Only after that, was I able to add the full non-aggregated data for all the individual rides, making sure that filters were applied across all sheets. This was because Tableau Public was not very responsive when trying to render all the marks for the whole data set.

Result #

Here is a simple dashboard, showing maps of the cluster centres and the individual rides.

You can see which zones have the greatest number of rides during each time slot. Filters allow you to compare different time slots.

Normally, Tableau adjusts the bubbles sizes on a bubble map relative to the data returned after any filtering. In order to make the bubbles resize relative to the total number of rides in the data set, I had to add a couple of calculated fields:

Total Rides (Fixed): A level of detail (LOD) field to return the fixed total ride count for the data set,
Ride Count: A field for the default ride count aggregation per cluster that returns the number of rides after applying filters,
Relative Rides = Ride Count / Total Rides (Fixed)

Final Thoughts #

The less dense clusters will be harder to service because they would attract fewer drivers while covering larger catchment areas. This could result in longer rider wait times. It would be advisable to perform a hierarchical cluster using just the subsets from the sparsest clusters (centred on Linnamäe tee, Vana-Mustamäe tn, and Vanasauna tee).

About

Sat, 21 Sep 2024 17:14:15 +0800

Welcome to my personal site. Here you can find my online resumé along with unsolicited opinions on topics in data science and analytics.

I am a senior data professional with a Ph.D in Machine Learning, MSc. in Business Intelligence and about twenty years experience across a wide range of technologies and special projects. I also come with management and leadership experience and other non-technical skills such as workshop design and facilitation, teaching and training, and stewarding Agile development teams to deliver great work. You can read more about my professional work on my LinkedIn Profile

Outside of work, I am a huge fan of avant-garde, extreme and experimental metal music. I am a regular concert goer in the great metal music scene in Berlin, where I live. I am and also an admin/moderator of the LGBTQ Metalheads Facebook Page, where we support and represent individuals who don’t always feel welcome in their local metal scenes.

I relax by listening to music at home (audiophile, hi-fidelity) on top of seeing treasured live shows. I also love to engage with many forms of contemporary art (visual, performance, craft/making). I have a torrid time trying to improve my German but I still try. My guilty pleasure is Ru Paul’s Drag Race. I also try to keep up to date with trends in data science and machine learning but anyone working in this field knows how fast things are changing. I prepare my own meals as much as possible, exercise and socialise regularly, and take travel adventure trips (at least) once per year. And that’s all the time there is to fill well accounted for.

In Winter, I’m usually in Singapore, because it’s some kind of second home to me and because the weather in Berlin at that time of year is far from ideal.

Online Resumé

Sat, 21 Sep 2024 17:14:15 +0800

TLDR Profile #

Senior data professional with 20 years experience in technical and leadership positions. Most recently, working as Senior Data Scientist Engineer for startups in Berlin, Germany.

Building on a foundation in Management Information Systems (MIS) and Business Intelligence (BI), I’ve advanced my analytical skills through forecasting and predictive analytics into prescriptive tools that deliver actionable insights. My Ph.D. research in explainable ML helps build transparency and a high-level of trust in decision-support systems, which has always been my focus. Over the years, I’ve driven change and digital transformation in medium size enterprises. I thrive in forward-thinking organizations eager to drive impactful results, systems optimization and process automation by leveraging AI, ML, cutting-edge analytics, and data-driven strategies.

Technical Skills and Qualifications #

PhD in Machine Learning (specific area: algorithmic development for explainable AI, with three published papers)
Master of Science (with Distinction) in Business Intelligence
Bachelor of Science in Microbiology
Extensive, recent experience in prompt engineering experience and developing * ChatGPT-based products and services.
Good working knowledge of NLP techniques and handling unstructured/text data
Excellent knowledge of statistical and machine learning models for decision making.
20 years experience in database technologies (SQL, OLTP, OLAP, ETL, Data Warehouse, Star Schema, 3NF Schema)
Python Programming (8 Years), Pandas, Numpy, Scikit-learn, Scipy, Statsmodels, XGBoost, Fast API, dramatiq, Flask, Jupyter, Anaconda
R Programming (8 Years), Tidyverse and native.
Tableau (2 Years)
SAS/JMP (2 Years)
GCP (2 Years), Datastore, BigQuery, CloudStorage
AWS (2 Years)

Non-technical (Soft) Skills #

Managing high performance teams of up to 25 people
Controlling web and software development budgets of up to USD8M
Team player at all levels from C-Suite and Senior Management Team to operational staff
Proven track record rolling out Agile practices and coaching scrum masters and product owners
Experienced meeting/workshop facilitator (innovation, prioritisation, strategy, team building)
Experienced teacher/technical trainer
Native English speaker, fluent in Italian, A2 in German and French

Career History #

Senior Data Scientist Engineer - Xapix Software GmbH (trading as Autopilot) #

May 2023 - Present
Creating (from inception to deployment and maintenance) a ChatGPT-based, end-to-end service for updating product pages on Amazon Marketplace.
- Using dynamic prompt engineering and NLP based post-processing, the product automatically re-writes various sections of the product page, to include SEO keywords and phrases.
- Improves the search-engine results page placement of the products, immediately resulting in increased sales.
- Maintains a natural-sounding title, bullet points and description to remain appealing to the customer/reader.
- Everything achieved in one round of prompting. No dialogue/intervention required for a fully automated service.
- Successful in both English and German (a market niche).
- New service resulted in dozens of new customer trials and signings. Customers see demostrable uplift in Page Views and Conversions in the order of $10Ks.
- Maintaining the data warehouse in GCP Big Query for terabytes of data processed weekly comprising Amazon Seller statistics and financial data.
- Created and maintained a suite of Colab Notebooks for management team to conduct impact reporting and customer pricing models.

Data and Analytics Lead (and Co-Founder) - Vishwavidya Pte Ltd (trading as EduX) #

Feb 2021 - Apr 2023
Modelling, organising and storing all company data in GCP Datastore, GCP BigQuery, GCP Cloud Storage, Kafka event logs, and memcached lookup keys.
Collection and curation of publicly available data, and their transformation into document embeddings, data products and actionable business intelligence using Python scrapy, ElasticSearch, and JanusGraph/Gremlin.
Ideating and evaluating machine learning driven features to improve the customer journey/experience to maximise customer engagement.
Building a team of data professionals and fostering their understanding of the company and data strategy (hiring two data science graduates and a data engineering intern).
Establishing and maintaining the data policy and governance (PII segregation and anonymization).
Development of ML models to predict specific app user behaviours (goal completion models).
Automation of ML pipeline (parameter optimisation and evaluation/comparison of multiple models), and versioning repository for models and preprocessing modules.
Schema design and data model to support microservices architecture and lightweight, non-intrusive partner integration.
Data ingest framework from the global internet with scrapy to our GCP DataStore transactional system and JanusGraph for graph analytics.
Automated document reader, with open source passport AI library and python multi-processing.
Combining BQ and GA into KPI’s, management info, and engagement analytic dashboards in Looker Studio.
Custom Full-text search indices for end users, pattern matching and partner integration.
Short text cleansing; a process for cleaning up snippets and captions from externally sourced text using (n-grams and word-embeddings, best regex discovery using grammatical evolution)
LDA Similarity and/or graph-based search engine enhancement (currently using ElasticSearch full-text indexing capabilities)
Social graph to track faculty and research contributions, determining which universities have prestige and social capital in a particular discipline

Freelance Analytics Consultant + Ph.D Research + University Lecturer + Startup Coach and Facilitator - Singapore, Sydney, London and Birmingham #

May 2016 - January 2021
Research leading to published work: CHIRPS: Explaining random forest classification, Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences, gbt-HIPS: Explaining the Classifications of Gradient Boosted Tree Ensembles.
Developing expertise various XAI methods and black box ML models (SHAP, LIME, Random Forests, Adaboost, XGBoost, and Random Forests, Extreme Pruning, Generalized Linear Models, Survival Models, Multi-state Life-cycle Models, Process Mining, Association Rules and Data Mining, Population-based search and modern optimization).
Automation of complete ML pipeline from pre-processing through parameter optimisation, and evaluation.
Generating flexible, dynamic, print-quality graphical results in Python (matplotlib, seaborn), R (ggplot, lattice and base graphics), and LaTeX (tikz).
Communication and public speaking, delivering seminars and lectures.
Research skills - reviewing complex academic literature to find gaps and opportunities.
Designing and executing experiments to test potential solutions and assess the results using suitable metrics and statistics.
Teaching Business Intelligence and tools (Tableau, SAP Lumira, SAS/JMP), Data and Statistics, designing curriculum including specialist workshops in Visualising Categorical Data.
Data Wrangling and building pipelines in Alteryx, KNIME and RapidMiner.
Developing an App engagement strategy using the Hooked model.
Delivering training on Project/Programme/Portfolio management, and Agile delivery.

Senior Business Operations Analyst - Kaplan Singapore #

Jan 2015 - Apr 2016
Growing the newly created cross-functional BA team.
Using R programming, Process Mining and tools (ProM, Fluxicon Disco), and setting up an OLAP (star schema) Data Warehouse with Tableau to identify performance trends and sources of inefficiency and proccess non-conformance, and using prescriptive analytics to propose improvements.
Liaising with C-Suite and Senior Management Team, to design a Benefits Realization process, allowing department heads to confirm that delivered change yielded intended benefits, such as redeployment of existing human resources to more impactful work.

Interim IT Director (6 Week Secondment) - Kaplan Singapore #

May 2014 - June 2014
Continuity of IT operations and projects following sudden resignation of existing IT Director.
Introducing Scrum and Kanban.
Coaching and mentoring IT team, reviewing personal development plans and reducing retention risk.

Progamme Director (Technology Projects) - Kaplan International Colleges #

May 2013 - Dec 2014
Worked with the CEO to create project portfolio aligned to her strategic objectives.
Created a benefits realisation plan to see projects through beyond delivery, ensuring that projected efficiencies and capabilities were fully realised and tracked to the bottom line.
Identified key stakeholders within the business to act as project sponsors, to help evangelise the benefits of each programme and drive their adoption throughout the business.
Communicated technical concepts to non-technical users in plain English.
Introduced the SMT to a variety of techniques for planning and executing strategic change (Scrum and Kanban, Balanced Scorecard, Portfolio & Programme Management, Innovation Games, Sketching at Work, Systems Thinking, Design Thinking, Business Model Canvas).

Earlier Career - Kaplan International, Kaplan Aspect, Aspect Education #

2002-2012
Retained through four acquisitions and changes of ownership, and regularly promoted.
Responsible for $2M software development budget.
Growing the software development team from 4 to 25 pax.
Creating and monitoring the full software development life-cycle management process.
Implemented and evangelised Agile, Scrum and related practices throughout the organisation, including to non-technical team, long before Agile was fashionable.
Implemented a Sarbanes-Oxley (SOX) compliance framework and all supporting processes, taking the IT division from zero to full compliance in a single quarter. Maintaining a range of in-house developed business software (MS SQL Server and VB) with no prior experience after the sudden departure of the principal developer.
Installation and support of ERP and Management Information Systems into 5 international offices across Europe.

Analysing SaaS Trial to Subscriber Conversions - Part 3 - Time Dependent Variables

Mon, 13 Nov 2023 00:00:00 +0000

Introduction #

You can read the Series Introduction here

In the previous post, we saw how survival curves can be created for different strata, or factors/categories of the independent variables, giving us a way to determine whether there are significant differences in the median survival time (or other quantile) between groups. The groups are fixed throughout the trial period. In a clinical or randomized control trial, these would be set as part of the experimental protocol. In this observational setting, these are often customer segments, such as industry vertical and other things that may not be under our control.

In this third and final post of this series, we use the Cox Proportion Hazards method to model the effect of variables that change over time. In this scenario, these variables are more likely to be things that are under our control, or things we can at least intervene on when we detect a desirable or undesirable effect.

For example, here we will see an analysis of cumulative customer logins, cumulative logged in time, and the number of distinct core features used. These are factors we can potentially nudge through incentives or gamification (although it will be hard to make strong causal inference about such complex interactions). We also see the effect of onboarding events and customer success outreach calls. These are very much under our control, in terms of the consistency of our customer success team to carry them out, and even the content of the calls and interaction with the customer.

The proportional hazard model tells us how these time-varying covariates influence the instantaneous risk of customer churn at any given moment, while accounting for their cumulative effects over time. Unlike the survival curves from our previous analysis that showed us static group differences, the Cox model quantifies the dynamic relationship between customer engagement behaviors and churn risk as both evolve throughout the customer lifecycle.

Most importantly, the model provides hazard ratios that translate directly into actionable insights: for every additional core feature a customer uses, or each additional customer success call completed, we can quantify the percentage change in churn risk. This allows us to prioritize interventions based on their estimated impact and helps us understand not just whether these activities matter, but by how much - giving us the foundation for data-driven customer success strategies and resource allocation decisions.

Theory Behind the Cox Proportional Hazards Estimator #

The model estimates the values for the coefficients $\beta_1, \beta_2, \dots, \beta_P$ for $P$ predictor variables where the instantaneous hazard function for individual $i$ is:

$$ h_i(t) = h_0(t) \times e^{(\beta_1 X_{1i} + \beta_2 X_{2i} + \dots + \beta_P X_{Pi})} = h_0(t) \times e^{(\boldsymbol{\beta}^T \cdot \boldsymbol{X}_i)} $$ The result is always interpreted as the ratio between two individuals, making it unnecessary to estimate $h_0(t)$. For example, if we have just one variable $X_1$ is the cumulative logged in time and customer A tallies up 20 hours more than customer B then we have a hazard ratio of:

$$ \frac{h_0(t) \times e^{\beta_1 * 20}}{h_0(t) \times e^{\beta_1 * 0}} = e^{\beta_1 * 20} $$ Here, the hazard ratio is $e^{\beta_1}$ which is equivalent to a multiplier of $\beta_1$ per additional hour logged relative to another individual.

Modeling #

We proceed to fit a naïve model with all our variables. The time-independent customer variables are also included, and the result table only shows the variables that are significant to a 95% confidence level.

variable	estimate	conf.low	conf.high	p.value	significance
homeware_industry	1.558641	1.234983	1.967121	0.0001861	***
content_channel	1.378438	1.075660	1.766443	0.0112022	*
high_initial_engagement	1.372458	1.017359	1.851501	0.0382071	*

We can see that the only significant items that show here are already accounted for by the stratification analysis in our previous post.

We can check this model assumptions that the hazard ratio is constant over time. Again, we just show the results that do not meet this assumption with 95% confidence.

variable	chisq	p
recent_activity	5.792295	0.0160966
high_initial_engagement	4.448864	0.0349245
content_channel	4.713107	0.0299335

We can see that two of our stratification variables violate the assumptions and should be modeled as strata. The recent activity marker also has a non-constant hazard and should not be included in this model.

We proceed to apply a refined set of variables to be modeled.

## # A tibble: 9 × 6
##   variable                   estimate conf.low conf.high  p.value significance
##   <chr>                         <dbl>    <dbl>     <dbl>    <dbl> <chr>       
## 1 homeware_industry             1.55     1.23       1.96 0.000221 "***"       
## 2 cs_outreach_day3              1.24     0.951      1.63 0.111    ""          
## 3 cumulative_session_minutes    1.00     1.00       1.00 0.111    ""          
## 4 cs_outreach_day14             1.19     0.948      1.49 0.136    ""          
## 5 referral_channel              1.17     0.916      1.50 0.205    ""          
## 6 cs_outreach_day21             0.873    0.675      1.13 0.304    ""          
## 7 cs_outreach_day7              0.915    0.735      1.14 0.424    ""          
## 8 login_rate_7day               0.637    0.177      2.29 0.490    ""          
## 9 onboarding_calls_completed    1.02     0.852      1.22 0.842    ""

Analysing SaaS Trial to Subscriber Conversions - Part 2 - Comparing Groups with Stratified Analysis

Wed, 04 Oct 2023 00:00:00 +0000

Introduction #

You can read the Series Introduction here

In the previous post, we saw how a Survival curve can be reframed as a Conversion Momentum Curve (CMC), giving us a way to esitmate when (counting in days from the start of a free trial period) we will reach conversion rate milestone of e.g. 5%, 10%, or 15% of free trial users. This is useful information because anything we can do to shorten the free trial period for our SaaS product offsets our fixed operating costs with new revenue.

We saw that our overall CMC looks like this:

This curve shows us the cumulative conversion rate at time $t$ (days) and gives us a sense of how the conversion rate evolves over the time spent in the free-trial period.

In this follow-up post, we use the standard KM-estimator method to estimate a non-parametric CMC and analyse differences between groups according to industry, acquisition channel, and geographic sales region. In Survival Analysis, this is called stratified estimation and, with certain assumptions, we can make inferences and hypothesis tests about the differences.

We consider customer invariant data that is already decided at the start of the free trial period that we cannot intervene on. We can only react to it. We track many other customer variables, such as the frequency with which they login and the customer success calls that we conduct with them. These types of variables are time dependent, just like the conversion event itself. The time they take place can have a profound effect on increasing or decreasing the likelihood of conversion. This requires a different analysis, which we won’t cover in this post. Look out for the next one if you are interested.

Industry Verticals #

We also saw that there were some apparent differences in the raw numbers for the two main industry verticals in which we operate: Homeware and Industrial Components

industry	n	convs	conv_rate	med_time_all	med_time_conv
homeware	1589	296	18.6	44.8	22.7
industrial	815	94	11.5	49.7	22.5

Let’s see how these differences manifest in the CMC analysis:

This chart gives us a very clear indication that these two curves are different. Very early on, the event lines separate and the confidence intervals no longer overlap. The charting code runs a Log-rank test in the background and p-value of this test is displayed. For completeness, we can also show the full Log-rank test table, which is based on a $\chi^2$ test statistic:

## Call:
## survdiff(formula = surv_obj ~ industry, data = survival_data)
## 
##                        N Observed Expected (O-E)^2/E (O-E)^2/V
## industry=homeware   1589      296      257      6.04      17.7
## industry=industrial  815       94      133     11.62      17.7
## 
##  Chisq= 17.7  on 1 degrees of freedom, p= 3e-05

Acquisition Channels #

Previously, we also so some crucial differences in the conversion rates per customer acquisition channel. It is essential for us to analyse this more deeply and understand how to optimise our marketing spend.

The confidence intervals have been omitted because there is a lot of overlap, which is somewhat messy with four overlapping curves.

## Call:
## survdiff(formula = surv_obj ~ acquisition_channel, data = survival_data)
## 
##                                N Observed Expected (O-E)^2/E (O-E)^2/V
## acquisition_channel=content  454       89     71.3     4.390     5.374
## acquisition_channel=organic  770      119    125.7     0.362     0.534
## acquisition_channel=paid_ads 649       92    107.7     2.300     3.179
## acquisition_channel=referral 531       90     85.2     0.270     0.345
## 
##  Chisq= 7.3  on 3 degrees of freedom, p= 0.06

The results here are rather nuanced. With a p-value of 0.062, the evidence against the null hypothesis of no difference between the curves is borderline or weak. We also see the event lines crossing in some places, indicating either no difference, or that an assumption of constant proportion between the hazard rates over time is violated.

Nevertheless, we can see visually that the content channel does diverge from the other three more dramatically when $t>40$, which supports what we saw in the raw numbers. In a situation like this, it can be more valuable to run a pairwise comparisons test.

## # A tibble: 6 × 4
##   Channel1 Channel2 p_value p_adjusted
##   <chr>    <chr>      <dbl>      <dbl>
## 1 organic  paid_ads  0.463      0.463 
## 2 organic  content   0.0457     0.137 
## 3 organic  referral  0.44       0.463 
## 4 paid_ads content   0.0094     0.0564
## 5 paid_ads referral  0.151      0.301 
## 6 content  referral  0.260      0.389

These results suggest that the differences between content vs paid ads and content vs organic channels are significant.

Industry and Channel Interactions #

We can, of course cross-tabulate and stratify on both items to see differences in channel performance across vertical.

## # A tibble: 8 × 6
##   industry   acquisition_channel     n conversions conversion_rate median_time
##   <chr>      <chr>               <int>       <dbl>           <dbl>       <dbl>
## 1 homeware   referral              348          74            21.3        42.1
## 2 homeware   content               316          67            21.2        40.5
## 3 homeware   paid_ads              429          77            17.9        47.1
## 4 homeware   organic               496          78            15.7        52.8
## 5 industrial content               138          22            15.9        38  
## 6 industrial organic               274          41            15          45.5
## 7 industrial referral              183          16             8.7        50.8
## 8 industrial paid_ads              220          15             6.8        64.6

The summary table in the previous post (reproduced here) showed some really obvious differences for the nested results and we had a rational business explanation for this situation: Paid search and referrals do not penetrate well to the right kind of customers in the Industry vertical, who seem to be more intentional in their search for our product.

Let’s flip back to a traditional orientation of the survival curve to understand the impact of this on our operations.

Here we see the conversions for referrals end after around $t>30$ and for paid search after $t>40$. The event line flattens out entirely. With this knowledge, we can adjust our offer to customers acquired through these channels and shorten the free trial period, cutting these operating costs much sooner.

Another perspective that we looked at in the previous post was conversion rate milestones. That is, an estimate of the free-trial period that will yield a given conversion rate. This is an adapation of the median survival time to use different quantiles. For example, the 0.1 quantile gives the number of days at which we still have 90% of the trial users unconverted. NA shows where the population never drops below this level.

industry	acquisition_channel	time_to_10pct_conversion
homeware	content	22.8
homeware	organic	23.8
homeware	paid_ads	24.8
homeware	referral	18.8
industrial	content	32.3
industrial	organic	22.8
industrial	paid_ads	NA
industrial	referral	33.5

In the above table we can see that the content marketing channel in the industrial vertical doesn’t look any better than the referral channel but we know this is not the case from the survival curve. In fact these two curves cross after the 30 day mark, demonstrating very different behaviours with many more customers converting after a longer period in trial.

We can also consider flipping this milestone to an estimate of the conversion rate or proportion of unconverted (survival) after specific periods in free-trial. This is useful to check how much value is left “on the table” when we end the trial periods too early.

industry	acquisition_channel	day_14_survival	day_30_survival	day_45_survival
homeware	content	0.97	0.84	0.77
homeware	organic	0.97	0.88	0.84
homeware	paid_ads	0.95	0.89	0.80
homeware	referral	0.96	0.84	0.77
industrial	content	0.97	0.92	0.80
industrial	organic	0.96	0.87	0.83
industrial	paid_ads	0.99	0.94	0.92
industrial	referral	0.99	0.91	0.90

Here we see that waiting until 45 days delivers an additional 10% conversion for this channel. This delivers very significant business value!

Geographic Sales Region #

Let’s take a look at the last customer invariant, which is of limited interest because there aren’t many levers we can pull around the way cultural and local business factors affect purchasing behaviours.

We see most of the regions are pretty similar, other than the German (DE) market which begins to lag behind the others after 30 days. Australia is also interesting because it picks up pace around that time and crosses (overtakes) all the other regions conversion rates to take the top spot by the end of the ten week cut off. People in this market really take their time to become familiar with the product for the first month or so and do not rush into making a decision but the product is quite popular in the end.

Let’s run the pairwise Log-rank tests to look for significant differences.

## # A tibble: 15 × 4
##    Region1 Region2 p_value p_adjusted
##    <chr>   <chr>     <dbl>      <dbl>
##  1 UK      CA       0.742       0.972
##  2 UK      US       0.565       0.972
##  3 UK      FR       0.953       0.972
##  4 UK      AU       0.942       0.972
##  5 UK      DE       0.0296      0.227
##  6 CA      US       0.972       0.972
##  7 CA      FR       0.797       0.972
##  8 CA      AU       0.688       0.972
##  9 CA      DE       0.196       0.589
## 10 US      FR       0.657       0.972
## 11 US      AU       0.531       0.972
## 12 US      DE       0.0538      0.227
## 13 FR      AU       0.884       0.972
## 14 FR      DE       0.0605      0.227
## 15 AU      DE       0.038       0.227

We see that all the comparisons with DE are significantly different when considering the non-adjusted p-values but not when the tests are adjusted for the multiple comparisons. No other significant differences are found. The insights gained may be used to modify incentives within the German sales process.

Customer Engagement Level #

As mentioned, this variable represents the initial feature selected by the customer during their initial onboarding. This is an ordinal factor with four levels because we don’t go so far as analyzing which of the four main features are activated. In fact, we simply convert this to a binary factor for Low Engagement (< 3 features) or High Engagement (>= features).

We see that the customers with a high initial engagement have a significantly higher conversion rate, certainly from around day 15 onward. Note that we cannot say with any certainty what the causal direction is. Are these customers already interested in all the features and are therefore more likely to convert, or does early exposure to all the feature convince them. It is not within the capability of this type of analysis to say with any certainty.

The Log-rank test result is also shown.

## Call:
## survdiff(formula = surv_obj ~ engagement_level, data = survival_data)
## 
##                                         N Observed Expected (O-E)^2/E (O-E)^2/V
## engagement_level=High (3-4 features) 1436      272      228      8.51      20.5
## engagement_level=Low (1-2 features)   968      118      162     11.97      20.5
## 
##  Chisq= 20.5  on 1 degrees of freedom, p= 6e-06

Summary and Conclusion #

In this post, we used the KM-Estimator to estimate a “flipped” survival curve, effectively $1-S(t)$. This is more intuitive when discussing cumulative free-trial conversions. We carried out the technique to stratify our curves by multipled customer invariant data. These are factors that are already determined before the trial and won’t change, such as the geographich sales region and the customer acquisition channel.

We used a few additional techniques to estimate milestone quantiles: by when will we achieve a certain conversion rate? how much does the conversion rate change between two fixed time points, such as 30 days and 45 days.

We used the Log-rank test (adjusting for multiple comparisons if necessary) to assess statistical significance between the strata.

Over all, with this very straightforward analysis, we were able to gain a lot of business insights that have the potential to reduce operating costs and boost conversions in certain markets.

Analysing SaaS Trial to Subscriber Conversions - Part 1 - Going Beyond the Binary Outcomes with Survival Analytics

Sat, 16 Sep 2023 00:00:00 +0000

Series Introduction #

This is part one of a series on using Survival Analysis techniques for Product Management. Survival models excel at analyzing time-dependent events where timing matters as much as the outcome itself. Unlike time series analysis that tracks metrics evolving over time (and requires data collected at regular time intervals), survival analysis is event-based and asks a different question: when will something happen, and what influences that timing?

The applications span the entire product lifecycle. A few examples include:

predicting feature abandonment during onboarding
modeling churn patterns for lifetime value calculations
optimizing A/B test duration
analyzing subscription upgrade timing
determining optimal re-engagement moments

Each scenario involves understanding not just whether an event occurs, but when it’s most likely to happen and what accelerates or delays that moment.

What makes survival analysis particularly elegant is how it handles incomplete information. When users haven’t yet converted or churned, traditional analytics sees missing data. Survival models see valuable information about persistence and decision-making patterns. The framework naturally accommodates both completed journeys and those still in progress at the end of the trial period, extracting insights that conventional funnel analysis simply cannot capture.

In part one, we’ll introduce a common scenario in product management; the trial-to-conversion journey. With the context in place, we’ll look at essential theoretical fundamentals and end this post with the first step of a much deeper analysis.

Future posts will build on this foundation with stratified estimation and comparison between the groups, and go on to advanced use cases such as time-dependent analyses that track how changing customer behavior, and external events and interventions can influence conversion probability in real-time.

Introduction #

Survival analysis has morbid origins – tracking ’time to death’ in clinical trials. The terminology feels grim: “hazard rates,” “time to failure,” “censored observations.” But flip the script to ’time to success’ and these insights become pure gold for product teams.

The mathematics stay the same, but the mindset transforms everything:

Customer drops out early? That’s not missing data – it’s product-market fit intelligence
Converts on day 40 vs day 10? That signals problems with onboarding effectiveness and operational costs

Traditional analytics asks “who converted?” Survival analysis asks “when do they convert, and what accelerates that moment?”

The difference is profound. You can separate controllable factors (onboarding sequences, CS outreach timing) from uncontrollable ones (industry vertical, company size), revealing the critical difference between customers who need more time versus those actively walking away. These insights unlock operational efficiency that directly impacts your bottom line.

As usual, the R code held out for brevity and echoed at the end of the post.

Just Enough Theory #

As usual, I want to give a sense of what’s happening under the hood without drowning in theory. Here’s a very brief primer on how Survival Analysis works.

The Survival Function: S(t) #

The survival function represents the probability that an event (in our case, conversion) has not occurred by time t:

$$ S(t) = P(T > t) $$

where $T$ is referred to as the time to failure in traditional Survival Analysis but in out context is the time to conversion. For trial customers, $S(t)$ tells us the probability that a customer hasn’t yet converted by day $t$. If $S(t)$ remains high for long, our trials are longer and we take a hit for ongiong operational costs of running the free tier and possibly losing out to the competition.

In our business oriented context, we’re rather more interested in $1 - S(t)$, the cumulative probability of success/conversion. I’ll refer to this flipped view as the Conversion Momentum Curve.

The Hazard Function h(t) #

This is the most theoretical element we’ll deal with but it’s useful to understand. The hazard function represents the instantaneous risk of conversion at time t, given survival (non-conversion) up to that point:

$$ h(t) = \lim_{\Delta t \to 0} \frac{P(t \leq T < t + \Delta t | T \geq t)}{\Delta t} $$ and it tells us the probability with which event $T$ takes place in the instant after time $t$.

Estimating the hazard function is essential for comparing strata - in our case, it could be customers’ industry vertical or acquisition channel. Given an assumption that the ratio between $h(t)$ for the strata remain constant over time, it is possible to make statistical claims about any differences.

The Cumulative Hazard (CumHaz) Function H(t) #

The cumulative hazard function represents the total accumulated risk (in our case of conversions, risk is a good thing) from the start of the trial up to time t:

$$ H(t) = \int_0^t h(u) du $$ In our business context, we can think of $H(t)$ as the Purchase Decision Momentum that has built up over the trial period. This helps us understand:

How conversion results accumulate over time.
Fluctuations in this conversion pressure, perhaps as a result of our customer success interventions.

The cumulative hazard provides a clearer picture of differences between customer segments than the instantaneous hazard rate alone.

Relationship Between Survival and Hazard #

There is a direct relationship that links Survival and Hazard functions via the CumHaz function, making it easy to switch perspectives:

$$ S(t) = \exp(-H(t)) $$

Non-parametric Estimators #

For distribution free estimation learned directly from data, there are methods for both $\hat{S}(t)$ and $\hat{H}(t)$ by counting at risk (not yet converted) units at the time $t$ of every event.

The Kaplan-Meier Estimator for Survival Function Estimation #

$$ \hat{S}(t) = \prod_{t_i \leq t} \left(1 - \frac{d_i}{n_i}\right) $$

where:

$d_i$ = number of events (conversions) at time $t_i$
$n_i$ = number of customers at risk just before time $t_i$

Kaplan-Meier Confidence Intervals #

The standard errors are calculated using Greenwood’s Standard Error formula:

$$ SE[\hat{S}(t)] = \hat{S}(t) \sqrt{\sum_{t_i \leq t} \frac{d_i}{n_i(n_i - d_i)}} $$

and these values are stabilised into the zero-one interval using Log-Log transformation:

$$ \hat{S}(t)^{e^{(\pm z_{\alpha/2} \cdot SE[\ln(-\ln(\hat{S}(t)))])}} $$

The Nelson-Aelen Estimator for CumHaz Function Estimation #

We can estimate the CumHaz function directly as well.

$$ \hat{H}(t) = \sum_{t_i \leq t} \frac{d_i}{n_i} $$

Scenario: Analysing SaaS Trial-to-Subscriber Conversion #

When you’re running a SaaS business with a 6-week free trial, the question isn’t simply “did they convert or not?” It’s far more nuanced.

At our e-commerce customer engagement platform, we help online retailers track their customer behavior, optimize product recommendations, and boost lifetime value. This all happens on a per-product basis. That is, we count each enrolled product as a conversion, while many products can come from each billed customer.

So we’ve become pretty astute at analysing our own trial-to-conversion processes. And this revealed something crucial: timing matters as much as the outcome itself.

Every day that a prospect remains in trial without converting carries real costs. Our infrastructure supports thousands of free-tier users analyzing their customer data, our customer success team schedules personalized demos and sends targeted outreach.

This business model has consequences; delayed conversions mean delayed revenue in a competitive market. When a retailer finally converts after 5 weeks instead of 2 weeks, we’ve absorbed additional operational costs while potentially losing deals to competitors who moved faster. Traditional conversion analysis treats late converters the same as early ones, but from both a cost and revenue perspective, they’re fundamentally different.

Using Survival Analysis on this data provides invaluable insight. When prospects go inactive, this isn’t missing data, it’s critical information. When a user actively unsubscribes, we know we lost out to a competing risk that has a time-critical element we can quantify. Most critically, just as you would expect with a logistic model, we can isolate the factors we can control (customer success outreach timing, onboarding flows, feature recommendations) from those we can’t (industry vertical, company size, acquisition channel), with the added benefit of understanding the element of timing.

The result? Actionable insights that directly impact our bottom line and operational efficiency.

What the Data Tells Us at First Glance #

Let’s follow a few good practices and get a sense of the key signals and early insights from our data.

Data Quality #

`1`	`## Total customers in trial: 500`

`1`	`## Total products in trial: 2404`

`1`	`## Time range: 5.3 to 70 days`

`1`	`## Any missing values: FALSE`

`1`	`## Median trial duration (all): 46.7 days`

`1`	`## Median trial duration (converted): 22.6 days`

The data is collected for a random selection of 500 customer accounts who started on the free trial at least 10 weeks prior to today. This means all products that don’t convert will ultimately be censored at day 70 no matter the exact date they started. As part of our regular operations policy, we also put accounts on ice after 6 weeks or 42 days of inactivity to prevent accruing operating costs for mothballed accounts. This cutoff results in earlier censoring for those records. Customers may also unsubscribe altogether. While rare, this event counts as a competing risk:

They chose one of our competitors.
They do not see the benefit of using the product.

We can see from the time range that the first conversion within this selection took place after 5.3 days. Our median trial duration date of 46.7 reflects the sampling parameters of the analysis.

Overall Conversions #

`1`	`## Overall conversion rate: 16.2 %`

16.2% is close to our prior expectation and is a pretty healthy conversion rate for any SaaS. However, given our niche, we’d hope to do even better with customers in active trials. So we need to deliver a set of actionable insights when we get into the full analysis.

Let’s keep exploring with simple groupings to get a sense of some key drivers.

Conversions by Industry Vertical #

We operate in two industry verticals (homeware and industrial components), so let’s take a quick look for any major differences:

industry	n	convs	conv_rate	med_time_all	med_time_conv
homeware	1589	296	18.6	44.8	22.7
industrial	815	94	11.5	49.7	22.5

We can see from this table that industrial components customers convert a bit less frequently, but there doesn’t seem to be a difference between the median conversion times. For both metrics, we’d want to find out if these differences are significant, and whether there are any drivers we can control to boost the weaker metric in each case.

Conversions by Customer Acquisition Channel #

We put a lot of product management and marketing effort into content marketing. This is to meet our customers where they are and show by example how our product solves some very specific pain points. There is a much greater CAC and opportunity cost on our content channel, so knowing whether this effort pays off is essential.

acquisition_channel	n	convs	conv_rate	med_time	med_time_conv
content	454	89	19.6	40.0	25.4
organic	770	119	15.5	50.3	22.2
paid_ads	649	92	14.2	52.7	22.7
referral	531	90	16.9	44.5	20.9

Here we see the conversion rate on the content channel is reassuringly a couple of percentage points higher than other channels. However, the time in trial is slightly longer. This is an interesting signal in the data and if we can clearly identify a driver, we would act. On the flip side, we wouldn’t want to interfere and reduce the effectiveness of this channel.

We also note that paid ads has the lowest conversion rate, while the CAC is also relatively high. This is worth a closer look too.

Crosstabs #

Let’s create a crosstab from the industry vertical and channel dimensions to get a more detailed understanding.

Conversions #

industry	content	organic	paid_ads	referral
homeware	21.2	15.7	17.9	21.3
industrial	15.9	15.0	6.8	8.7

It seems that industrial customers acquired by paid ads and referrals are the least likely to convert, while these same channels perform very well in the homeware vertical. This early insight aligns pretty well with the product manager’s domain experience: these customers have very different profiles. Industrial customers that convert were more directed in their research and didn’t tend to click on sponsored links. Similarly, their procurement processes are stricter and more hierarchical. Network effects are less likely to break through.

Median Time to Convert #

industry	content	organic	paid_ads	referral
homeware	23.5	22.2	24.2	20.6
industrial	32.2	22.1	21.1	21.7

We’re also seeing a really long median conversion time relative to the other groups for industrial/manufacturing customers who were acquired through content marketing. This was not at all expected but we start with a working hypothesis that these customers are much more deliberative in their decision-making, given the depth of engagement early in the funnel.

Visual Summary #

As always, a picture tells a thousand words when it comes to data analysis. Here we see an intuitive view of these preliminary EDA findings

Other Metrics #

We have a few other dimensions such as customer region (we sell into six different countries), and the initial engagement level in the trial. Then there are time dependent metrics, such as ongoing engagement. To keep this post from ballooning, we’ll get more into those weeds in the forthcoming posts.

Tracking the Customer Journey: From Trial Day One to Decision Day #

Now we move from exploratory static snapshots to dynamic storytelling using Survival Analysis. The classic Kaplan-Meier estimation method lets us track how conversion probability evolves day-by-day throughout the trial period.

Reframing the Survival Narrative With Overall Conversions #

We’ll look first at a traditional survival curve, which actually measures the probability of not converting. The survival curve has a few key features. It always begins at 1.0, with all individuals in the study at risk. It is monotonically falling because once an individual converts, they never rejoin the at-risk pool. The small vertical bars on the curve represent conversion and censoring events. On the above curve, they are rather dense but we’ll see later with stratified curves that this can be more informative.

However, I want to flip these plots upside-down to create conversion momentum curves, showing a more intuitive cumulative probability of conversion over time. We also don’t want to talk about the number of individuals still “at risk” but rather the number of products “yet to convert.”

This reframing transforms survival analysis from a clinical mindset into a growth mindset.

As we go deeper with the investigation, we will stratify the curves to tell the story of how different customer segments build momentum toward their purchase decision, revealing not just who converts, but when conversion accelerates and what drives that critical moment of commitment.

Redefining Meaningful Milestones: From Medians to Early Indicators #

Traditional survival analysis relies heavily on the median survival time. This is the point where 50% of the population has experienced the event. It’s a natural milestone that splits the data in half and provides an intuitive benchmark for comparison.

But here’s the challenge: our SaaS trial data shows only 16% conversion rates, meaning the survival curve never drops below 84%. The median simply doesn’t exist in our dataset – mathematically.

This reality forces us to adapt our approach. We can focus on the quantiles that actually occur within our business timeframe – the 5th, 10th, and 15th percentiles of conversion timing. These early milestones capture the behavior of customers who do convert, revealing actionable insights about when momentum builds and decisions crystallize, rather than chasing statistical conventions that don’t match our business reality.

quantile	days	lower	upper
0.05	16.5	15.3	18.5
0.10	24.2	22.6	26.9
0.15	35.4	33.2	41.4

This is a bit of a mouthful to explain, but in essence, it’s just a question of drawing a horizontal line from the required conversion rate, and reading the value at the x-axis where the line intersects the curve (and it’s confidence intervals). This is easier to understand with the following annotated plot.

Summary #

We’ve introduced Survival Analysis as an extremely useful tool in the Product Management life cycle. We covered the fundamental theoretical principles, including non-parametric estimation (learning from timed event data). Then we introduced a specific scenario; trial-to-conversion, explored our data set and took the first step in modeling and interpreting the Survival curve, through the lens of Product Management.

In the next post, we’ll estimate new survival curves, stratified by our key factors, such as vertical and channel, to compare Survival estimates between groups.

Educational Attainment in England - A Deeper Dive

Wed, 30 Aug 2023 00:00:00 +0000

Introduction #

On the 25th July 2023, the UK Office for National Statistics produced a wonderful piece of data journalism with open access to their dataset measuring educational attainment across England. The article’s title “Why do children and young people in smaller towns do better academically than those in larger towns?” hides a bold claim in the form of a question. I like to assume that their research question(s) did not start from such a knowledge claim but rather the title emerged from the themes they discovered during their investigation.

In fact, the article’s subtitle provides more of a clue over the sort of questions they must have been asking, uncovering correlations to attainment with factors in the social environment:

"Town size, income deprivation and the education level of other residents are all related to the educational attainment of pupils in English towns."

Overall, I found the article fascinating and enjoyed the approach they took, with plots and visuals that offer immediacy of comprehension to a non-technical audience, and explanations that are free of statistical jargon.

Grateful that the authors have published the data set along with embeddable chart objects, I thought it would be interesting to consider some deeper analyses with a more technical audience in mind. So, I’ve re-run some of the key findings, adding a handful of more advanced statistical tools. These tools allow me to make more detailed inferences, and possibly deeper insights.

Obviously, the ONS do not need me to tell them how to analyse data! I’m sure that the ideas presented here were considered, discussed, and then pared back to the most digestible form. Part of the ONS remit is to choose a level of detail to communicate to the broadest possible audience. My intention with this write up is simply to work in a way that is not constrained by the same concerns.

Note, as usual, I defer printing the source code until the end of the document, except where I have modified the data. In that case, the code is shown so the reader can understand the actions in context

Note also, where I have embedded plots from the ONS article, you will see the appropriate credits. Everything else here is my own.

Research Questions #

The ONS investigation uncovers some important interactions between town size and level of income deprivation, but the complex nature of this effect is set aside in favour of a series of easier to understand bivariate analyses. I would like to consider the following:

Is town size a factor in educational attainment when controlling for other variables, such as level of income deprivation?
Are regional effects a significant factor in educational attainment, or are there more powerful local factors at play?

Analytic Strategy #

I will proceed in two phases:

An initial exploratory analysis with descriptive statistics, undertaking an enhanced approach based on the original article:

A critical assessment of the descriptive plots and charts inn the original report, adding a further level of detail, where possible.
An audit of the differences between groups that were shown in the original article, engineering new features from the provided statistics that offer a greater level of statistical scrutiny (modeling variance, uncertainty, etc).

A modelling approach that handles hierarchical groupings such as town and region, enabling us quantify the differences in attainment at different geographic levels.

Initial Exploratory Analysis #

I have downloaded the original full data set. The data refer to a cohort who sat GCSEs in 2012-2013 and is a longitudinal study of their performance from Key Stage 2 (2007-8), and comprising GCSE results, and qualifications and/or other indicators at ages 18 and 22. That is to say, all the counts, percentages and scores listed relate to this one cohort, unless otherwise stated. Where something is listed with the word “score” in the field name, this is a derived score for which the formula may or may not be available. However, it is applied across the cohort and can be used for between group comparisons.

The data are imported from a csv file. There are 1104 rows and 33 columns. All the necessary detail is in the original article. So, for brevity, I will skip over a comprehensive EDA and descriptive statistics. Instead, I will look at some of the original visual analyses and the claims made about them, and then carry out an enhanced analysis.

“Young People Do Better Where The Older Generations Have Higher Attainment” #

The article presents findings of an inter-generational relationship, which stands up to instinct and reasoning. It is fair to assume that educated parents, other family members and friends are better equipped, through their own experience, to support their children’s education more deliberately and methodically.

The main dataset, in fact, contains a categorical variable for older HE attainment, and here I present one of the first exploratory plots I created from it, as an alternative view of the same information. This is a one-sided bee swarm plot of the cohort attainment score, coloured by the older HE attainment category.

I like this plotting approach when there are enough data points over the range of given values because the shape gives a sense of the data density. A kernel density estimation (KDE) plot is a standard tool to use in the visualisation of univariate data but there are subtleties that are good to view with this alternative perspective. Consider the following two plots for comparison:

You can see with the KDE plots we still visualise the distribution among the factor levels, but the emphasis is on their separateness. The bee swarm plot, on the other hand, gives and impression of the distribution while allowing the data points to mix. It’s more intuitive than scientific but given the subject matter is families and communities, it seems to offer a better metaphor for the towns and villages from where the data are drawn.

Reproducing the original scatterplots #

The raw percentage of older graduates in the population was available as a separate file download in the article. The original authors presented a scatter plot of this relationship, showing a strong correlation. I have recreated this plot, annotating with the Pearson correlation.

“Pupils in Small Towns Do Better on Average” #

Now we move on to the claim that I am most sceptical about. The claim is supported by a beautiful beeswarm plot in the original article.

While it seems to be true that the groups preset with noticeable mean differences, these differences do seem to be very small. Furthermore, the variance for small towns is certainly the largest by quite some margin, and we see that there are many small towns scoring worst in the country. So, the sub-heading claim doesn’t appear to tell the whole story. To their credit, the authors do expand on this point for themselves and the claim itself appears to be sound under further scrutiny. For example, I can verify it through an ANOVA test of the three designations (Small/Medium/Large Towns), and a Student’s t test between just the Small and Medium Towns. Here are the results:

##               Df Sum Sq Mean Sq F value  Pr(>F)   
## size_flag      2    137   68.27   5.226 0.00551 **
## Residuals   1079  14097   13.07                   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
## 	Welch Two Sample t-test
## 
## data:  small_towns and medium_towns
## t = 2.3189, df = 758.57, p-value = 0.01033
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  0.1593041       Inf
## sample estimates:
##  mean of x  mean of y 
##  0.2966166 -0.2530946

For both tests, the p-value is well below the 0.05 significance level, supporting the main statistical claim about differences between groups. However, I am left with niggling concerns:

The variance for small towns is really large, with the extremes eclipsing the other groups.
I am also bothered about the fact that there be an inflationary effect of repeating samples from the same broader geographical area.

While the original report does dig into large geographical regions, such as the North West, or the South East, these are enormous designations that do not capture granular socio-economic realities neighbourinng of small towns.

I have a hypothesis that nearby and neighbouring small towns are very unlikely to be genuinely independent observations when it comes to issues such as income deprivation. What if all the towns in a close geographical area suffered similar issues after the loss of a major local employer, such as a factory, mine or call centre? If a cluster of nearby towns provide similar measures because they are related by geography, proximity and recent economic realities, then their shared misfortune provides a cluster of non-independent samples. As statisticians, we must be vigilant towards sampling issues like this. Such sampling issues are the greatest source of bias and error in statistical investigations.

I will return to this topic later on.

“Towns With High Educational Attainment Have Low Income Deprivation” #

This is the most expected and predictable claim and any rational thinking person would intuit a causal relationship: income deprivation contributes to low educational attainment. I note that the original article is worded in the inverse order than my former statement, and teh wording also lands on the least stigmatising category labels (high educational attainment, low income deprivation). I think this is probably a deliberate editorial decision, made to ensure that no one accuses the ONS of implying such a causal relationship when they can only provide statistical evidence for correlation. I make no such claim either but I’m happy to point out that a causal relationship is very likely what people would assume.

The relationship can be seen in another scatter plot of attainment score vs income score. There was an accompanying additional csv file (for whatever reason, the income score was not part of the main file). Here I annotate the plot again with the Pearson correlation.

We see here a heteroskedastic pattern (reversed), with variance growing as the attainment score falls. This pattern adds to my suspicion other factor(s) are at play, mediating the relationship to attainment in lower income areas.

“Small Towns Are Less Frequently Classified as Income Deprived” #

Given the findings so far, this claim about the nature of small towns seems extremely important. It notes the under-representation of income deprivation as town size decreases. This is an interaction effect that really needs a closer look to understand the drivers for difference in attainment. Think about it; if the sample of small towns contains many more higher income towns, this underlying disproportion will bias the mean attainment score upwards, given the previous correlation claim.

The proportion of each town size designation under each income deprivation class (Higher/Mid/Lower Income Deprivation) is shown in the article with a fixed width horizontal bar plot.

Such plots are useful for their immediacy but some information in lost. I find various kinds of strucplots much more illuminating, especially when there are two- or more-way interactions. I present two mosaic plots for illustration. The Expected frequencies plot shows the expected counts under the Null hypothesis that income deprivation and town size are independent. The second Actual frequencies presents the observed counts.

A mosaic plot is ideal when dealing with categorical and count data. It uses tile area to represent the count of each category intersection, resulting in tiles of varying width and height. The shading of the Actual frequencies mosaic follows a scheme according to the magnitude of the Pearson residuals (of a test of actual vs. expected under independence) at each intersecting level of the categories. The darkest blue indicates a significant over count and the darkest red is conversely a significant under count, when compared to the Null hypothesis. By linking shading depth to the Pearson residuals, the mosaic plot is more than just a visual check. It is a rigorous statistical test.

The mosaic plot provides evidence of the interaction that gave rise to my concerns and why I think it is essential to control for town size and income deprivation when investigating educational attainment.

“The Relationship Between Attainment and Income Differs Between Regions” #

In the original article, there is no mention of the potential effect of geographical proximity between towns and their recorded observations. I can understand that a choice was made not to dive into the increasing complexity of hierarchical modeling for the sake of holding the audience’s attention on the main findings. The authors present a very nice dot plot, illustrating the North West region’s best scores at all three income levels.

As mentioned before, I believe that geographical clusters of smaller towns may have economic effects in common that could bias the statistical inferences drawn from this data set. It would be prudent to find a way to adjust for the amplified signal arising from shared socio-political histories that are linked by physical proximity and connections between communities. Furthermore, the nearest large urban centre could also be an influence because of the presence of centralised education authorities and local funding decisions. To analyse attainment, taking into account the geographical hierarchy of nearest large urban centre and region, I will use a Linear Mixed Model (LMM).

When determining the modeling approach with LMM, it is important to make reasoned decisions about a plausible structure for the hierarchical relationship. LMM allow for different intercepts, or different intercepts and slopes for each independent variable. The proximity relationship is essentially a binary choice: Aylesbury either has London as its closest city, or it does not. Hull is in the region Yorkshire and the Humber, or it is not. Both of these examples are true, by the way. So, it only makes sense to model different intercepts here for both hierarchical factor levels. Slopes would be applied to numerical independent variables. The “/” notation indicates a hierarchical grouping here of ttwa11nm (urban centre) within rgn11nm (region).

lmer_model <- lmer(
  education_score ~ 1 + size_flag + income_flag + (1 | rgn11nm/ttwa11nm),
  data = english_education,
  subset = town_size_filter
)

summary(lmer_model)

## Linear mixed model fit by REML ['lmerMod']
## Formula: education_score ~ 1 + size_flag + income_flag + (1 | rgn11nm/ttwa11nm)
##    Data: english_education
##  Subset: town_size_filter
## 
## REML criterion at convergence: 5114.4
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.9102 -0.6342 -0.0349  0.6261  3.3471 
## 
## Random effects:
##  Groups           Name        Variance Std.Dev.
##  ttwa11nm:rgn11nm (Intercept) 1.1978   1.0944  
##  rgn11nm          (Intercept) 0.3904   0.6248  
##  Residual                     5.8334   2.4152  
## Number of obs: 1082, groups:  ttwa11nm:rgn11nm, 164; rgn11nm, 8
## 
## Fixed effects:
##                                    Estimate Std. Error t value
## (Intercept)                         -2.4851     0.3629  -6.848
## size_flagMedium Towns               -0.2876     0.3007  -0.956
## size_flagSmall Towns                -0.4828     0.2916  -1.656
## income_flagMid deprivation towns     2.2839     0.2263  10.091
## income_flagLower deprivation towns   5.8011     0.2068  28.051
## 
## Correlation of Fixed Effects:
##             (Intr) sz_fMT sz_fST in_Mdt
## sz_flgMdmTw -0.619                     
## sz_flgSmllT -0.632  0.825              
## incm_flgMdt -0.131 -0.094 -0.137       
## incm_flgLdt -0.103 -0.125 -0.257  0.477

There is a bit more to interpret than we would get from a multivariate linear model. Importantly, the model converges easily and the scaled residuals are fairly symmetrical about zero. The fixed effects only show correlations between different levels of the same factor. These are all good diagnostic indicators, as is the residual plot here:

The main statistics are really interesting. The intercept can be taken as the base value of attainment score and is set to reference levels “Large Towns” and “Higher deprivation towns”. The fixed effects of income deprivation agree with the official investigation. As the levels of income deprivation ease, the attainment outcomes rise dramatically. As I said before, this is the most obvious conclusion of the whole report.

However, in a classic case of Simpson’s paradox, the reported correlations between town size and attainment are reversed once we control for income and the geographical proximity in this hierarchical model. This is evident from the negative values associated with medium and small towns. The model shows that wealthy small towns do well while poor small towns do worst. The disparity in effect sizes explains the heteroskedastic scatter plot we saw earlier.

	coefficients
Large Towns / Higher deprivation towns	-2.4851
Medium Towns	-0.2876
Small Towns	-0.4828
Mid deprivation towns	2.2839
Lower deprivation towns	5.8011

Looking at the variation in the random effects, we can see that the regional differences are also present, as reported in the article but rather more of the variation is explained by the nearest urban centres (the standard deviation is approximately double). So, things are much as I initially suspected. This result seems to validates my intiution; clusters of towns in the same geographical area are subject to local effects that also contribute to their educational outcomes. This effect is stronger than the wider regional effect. This concept is not reported anywhere in the original report.

Group	Std.Dev.
ttwa11nm:rgn11nm	1.0944333
rgn11nm	0.6247878
Residual	2.4152362

`1`	`## $rgn11nm`

`1`	## $`ttwa11nm:rgn11nm`

Summary and Conclusions #

I used a publically available data set that was provided with an already comprehensive piece of data journalism and analysis, to see if I could uncover any deeper insights than those presented to the original article’s non-technical audience.

I confirmed the validity of some of the claims using standard statistical tests that were not discussed in the original write-up.
I showed some alternative visualisations for the data that provide more nuance and information than the simpler graphics that had been chosen for a less technical audience.
I used a hierarchical linear mixed model to control for multiple interacting factors and demonstrated that:
- proximal geographical effects are stronger than broad regional effects
- the original article may have made an erroneous claim about effect of town size on educational attainment because the effect was reversed when controlling for income deprivation

This detailed analysis shows that having multiple tools for different depths of analysis allows you to use multiple perspectives when looking for patterns and draw inferences from your data. Some tools suit a less technical audience and offer immediacy of understanding, while other more detailed tools are required for high impact decision support and policy making.

Appendix #

Here you can find the source code.

library(knitr)
library(tidytuesdayR)
library(magrittr)
library(dplyr)
library(tidyr)
library(ggbeeswarm)
library(vcd)
library(vcdExtra)
library(lattice)
library(survey)
library(ggplot2)
library(gridExtra)
library(forcats)
library(lme4)
library(MCMCpack)

opts_chunk$set(warning = FALSE
              , message = FALSE
              , echo = FALSE
              )

hook_output <- knit_hooks$get("output")
knit_hooks$set(output = function(x, options) {
  lines <- options$output.lines
  if (is.null(lines)) {
    return(hook_output(x, options))  # pass to default hook
  }
  x <- unlist(strsplit(x, "\n"))
  more <- "..."
  if (length(lines)==1) {        # first n lines
    if (length(x) > lines) {
      # truncate the output, but add ....
      x <- c(head(x, lines), more)
    }
  } else {
    x <- c(more, x[lines], more)
  }
  # paste these lines together
  x <- paste(c(x, ""), collapse = "\n")
  hook_output(x, options)
})

par(mar = c(4,3,3,1))
source("HeartTheme.R")
english_education <- as_tibble(read.csv("english_education.csv")) %>%
  dplyr::mutate(
      proportion_of_higher_ed=factor(
      level4qual_residents35_64_2011,
      levels = c("Low", "Medium", "High", NA)
    ),
      income_flag=factor(
      income_flag,
      levels = c("Higher deprivation towns", "Mid deprivation towns", "Lower deprivation towns")
      )
  )

income_and_edu <- read.csv("income_and_education_score.csv", sep = ";") %>%
  dplyr::select(-X) %>%
  rename(
  town=Town.name,
  attainment_score=Educational.attainment.score,
  income_score=Income.deprivation.score
  ) %>%
  mutate(
    income_score=as.numeric(sub(",", ".", income_score)),
    attainment_score=as.numeric(sub(",", ".", attainment_score)),
    town=sub(" BUA.*", "", town)
  ) %>%
  as_tibble()

digits_only <- function(vec) {
  re <- regexpr("[[:digit:]]{1,2}", vec)
  matches <- regmatches(vec, re)
  return(matches)
}

older_and_edu <- read.csv("education_and_older_attainment.csv", sep = ";") %>%
  dplyr::select(-X) %>%
  rename(
    town=Town.name,
    attainment_score=Educational.attainment.score,
    percentage_of_higher_ed=Residents.aged.35.64.with.Level.4.or.above.qualifications
  ) %>%
  mutate(
    percentage_of_higher_ed=as.numeric(digits_only(percentage_of_higher_ed)),
    attainment_score=as.numeric(sub(",", ".", attainment_score)),
    town=sub(" BUA.*", "", town)
  ) %>%
  as_tibble()
english_edu_scatter <- function(g, correlation, corr_x, corr_y) {
  g + geom_point(colour = myPal[4], alpha = 0.75) +
  geom_point(shape = 1, colour = myDarkCornFlower) +
  annotate("label", x = corr_x, y = corr_y,
           label = paste("Correlation:\n", round(correlation, 2))) +
  theme_bw()
}
size_flags <- c("Large Towns", "Medium Towns", "Small Towns")
town_size_filter <- english_education$size_flag %in% size_flags
shorten <- function(string, end_pos) {
  shortened <- substr(string, 1, end_pos)
  ifelse(nchar(string) > end_pos,  paste0(shortened, "..."), shortened)
   }
shorten_20 <- function(string) { shorten(string, 20) }
classes <- sapply(english_education, class)
full_names <- names(classes)
short_names <- sapply(full_names, shorten_20)
names(classes) <- NULL
names(short_names) <- NULL
notes <- c("Unique ID", "Raw Town Name", "Population (Census 2011)", "Size Category", "Region Name", "Coastal Category", "Coastal Detail Category", "Nearest Urban Centre ID", "Nearest Urban Centre Name", "Urban/Rural Category", "Job Density Category", "Income Category", "University Category", "Graduates Among Older Residents Category", "Count Students at Key Stage 4", "% Key Stage 2 Attainment", "% Key Stage 4 Attainment", "% Students at Level 2", "% Students at Level 3", "% 19yo in Full Time Higher Education", "% 19yo in Further Education", "% 19yo in Apprenticeship", "% 19yo With Earnings Above 0", "% 19yo With Earnings Above 10K", "% 19yo Jobless", "Ignore (most NA)", "% 22yo Highest Qual up to L2", "% 22yo Highest Qual L3-5", "% 22yo Highest Qual L6 and up", "Score 22yo Highest Qual", "Score Relative Attainment")
kable(cbind(short_names[1:31], classes[1:31], notes[1:31]))
ggplot(
  english_education,
  aes(y = 1, x = education_score, colour = proportion_of_higher_ed),
  size = 0.5) +
  geom_beeswarm(side = 1) +
  scale_y_discrete(
    expand = expansion(mult = c(0.02, 0.1))) +
  theme_bw() +
  theme(axis.text.y = element_blank(), axis.title.y = element_blank())
ggplot(
  english_education[english_education$proportion_of_higher_ed %in% c("High", "Medium", "Low"),],
  aes(x = education_score, fill = proportion_of_higher_ed, colour = proportion_of_higher_ed),
) +
  geom_density(alpha = 0.25) +
  theme_bw() +
  theme(axis.text.y = element_blank(), axis.title.y = element_blank())
ggplot(
  english_education[english_education$proportion_of_higher_ed %in% c("High", "Medium", "Low"),],
  aes(x = education_score, fill = proportion_of_higher_ed, colour = proportion_of_higher_ed),
) +
  geom_density(position="stack", alpha = 0.25) +
  theme_bw() +
  theme(axis.text.y = element_blank(), axis.title.y = element_blank())
with(older_and_edu,
  correlation_oe <<- cor(
      attainment_score, percentage_of_higher_ed, method="pearson"
    )
  )

g <- ggplot(
  older_and_edu,
  aes(x = attainment_score, y = percentage_of_higher_ed),
)

english_edu_scatter(g, correlation_oe, 7.5, 20)
summary(
  aov(
    education_score ~ size_flag,
    data = dplyr::filter(
      english_education,
      town_size_filter
    )
  )
)
small_towns <- english_education %>%
  dplyr::filter(size_flag == "Small Towns") %>%
  dplyr::select(education_score)

medium_towns <- english_education %>%
  dplyr::filter(size_flag == "Medium Towns") %>%
  dplyr::select(education_score)
  

t.test(small_towns, medium_towns, "greater")
with(income_and_edu,
  correlation_ie <<- cor(
      attainment_score, income_score, method="pearson"
    )
  )

g <- ggplot(
  income_and_edu,
  aes(x = attainment_score, y = income_score),
)

english_edu_scatter(g, correlation_ie, 5, 0.8)
size_income <- table(english_education[c("size_flag", "income_flag")])
size_income <- size_income[
  size_flags,
  c("Lower deprivation towns", "Mid deprivation towns", "Higher deprivation towns")
]
dimnames(size_income) <- list(
  size_flag = c("Large", "Medium", "Small"),
  income_flag = c("Lower", "Mid", "Higher")
)
expected_size_income <- round(independence_table(size_income), 0)

mosaic(
  expected_size_income,
  main="Expected frequencies",
  labeling = labeling_values,
  value_type = "expected",
  gp_text = gpar(fontface = 1),
  rot_labels = c(top = -20, left = 70)
)

mosaic(
  size_income,
  gp = shading_hsv,
  gp_args = myShadingPar,
  main="Actual frequencies",
  labeling = labeling_values,
  value_type = "observed",
  gp_text = gpar(fontface = 1),
  rot_labels = c(top = -20, left = 70),
  legend = FALSE
)
lmer_model <- lmer(
  education_score ~ 1 + size_flag + income_flag + (1 | rgn11nm/ttwa11nm),
  data = english_education,
  subset = town_size_filter
)

summary(lmer_model)
plot(lmer_model)
kable(cbind(c("Large Towns / Higher deprivation towns", "Medium Towns", "Small Towns", "Mid deprivation towns", "Lower deprivation towns"), coefficients = round(lmer_model@beta, 4)))
coeff_sd <- as.data.frame(VarCorr(lmer_model))[c(1, 5)]
names(coeff_sd) <- c("Group", "Std.Dev.")
kable(coeff_sd)
dotplot(ranef(lmer_model))[2]
dotplot(ranef(lmer_model))[1]

Tech Stock Stories - The Pandemic Winners

Tue, 07 Feb 2023 00:00:00 +0000

As we breath a collective sigh of relief that the COVID-19 pandemic is behind us, it’s interesting to think about the winners and losers of this unique period of living memory.

Judging by the stock performance of US tech giants, I’d say their shareholders did rather well while the rest of the world switched to remote work and spent all our time indoors.

Please enjoy my Tableau Viz below, which is a candle chart of the stock performance of US tech giants during the pre- and post-pandemic period. We can see the month average highs, lows, opens and closes, and you can pick the company of interest from the drop down list on the right.

The data set was presented yesterday as part of the Tidy Tuesday series, popular with R users.

What is Analytics Engineering?

Mon, 10 Oct 2022 00:00:00 +0000

The well-known roles of the data team are Data Analyst/Scientist and Data Engineer. Yet, in recent years, there has been a growing demand for data-driven decision making to be distributed throughout the organisation, with a traditional data team at risk of becoming a bottle neck as the growing need for insights and analytics cannot be fully met.

With this increased demand has come the emergence of a new role - the Analytics Engineer - for the typical data team to better encompass these evolving responsibilities, and specialist technologies. This evolution has led to a paradigm shift in data processes from Extract-Transform-Load (ETL) to Extract-Load-Transform (ETL), which facilitates a more self-service oriented Business Intelligence (BI) operating model where data analysts and data scientists can be more embedded with domain teams.

How does this new role fit in and solve the problem?

Traditional data analysts, data scientists, and machine learning engineers want to focus on analyzing and modelling data.

Data Engineers are responsible for designing and implementing robust, scalable and fault tolerant data pipelines to Extract data from homegeneous systems (operational, external, IoT/Edge etc) and Load them to a Data Lake or Data Warehouse. These raw data are rarely available in a form that is well-suited to analytical workloads.

The task to Transform these raw data into analytical data structures suffered greatly from a lack of separation of concerns. The Data Engineering ETL process was generelly implemented without taking domain knowledge into account. As a generic technical process, the resulting data structures were inflexible and not analysis ready.

The data analysts/scientists have to spend further time wrangling data into the format they want, and the rigid Data Warehouse architecture is not flexible enough for self-service BI for less technical users with high domain knowledge.

The result of this unclear separation of concerns is that the Transform step is notoriously hard to document and maintain. Data quality assumptions are spread over multiple systems. The steps required to rebuild modelled data are not owned nor executable by a single system. The lineage of data in the user-facing data sources is hard to trace. All these issues have a knock-on effect to debugging issues.

Analytics engineers step into this gap. Their focus is on turning raw data into the building blocks of clean, tidy, documented and discoverable data that enable these other roles to do their best work. The required skills for succeeding at this role are:

cloud scale data warehouse technology (e.g. GCP BigQuery, Snowflake, AWS Redshift)
technical proficiency at data manipulation/wrangling (expert SQL and data manipulation languages such as Python or R)
data orchestration tools such as dbt and Apache Airflow
version control
good domain knowledge and/or excellent stakeholder liaison (business analysis, requirements gathering)
monitoring data accuracy and freshness
excellent documentation and discoverability

By focusing on the technical aspects of data analytics and data transformation to meet the business objectives, analytics engineers release the full value of business data, as data analysts/scientists and non-technical data consumers can get on with the the work of extracting insights, building predictive models, and making better decisions. As the amount of data generated by organizations continues to grow, the demand for skilled analytics engineers is likely to increase, making this a promising career path for those interested in the intersection of data and technology.

Data Pipelines For Machine Learning

Tue, 25 May 2021 00:00:00 +0000

Data pipelines are essential for machine learning projects as they help to manage the flow of data from various sources, ensure data quality, and automate the process of data preparation.

What’s the end goal for a data pipeline? The resulting clean and processed data may be used for analysis, business intelligence and reporting. Another common use these days is as the input for Machine Learning (ML).

While data pipelines are often hand-coded in high level programming languages such as Java, there are plenty of configurable (point-and-click) tools available to do this task. One benefit of using such tools is that many of them have built-in components for completing the ML tasks of feature engineering, training, evaluating and deploying ML models. In this blog post, I will compare and contrast SAS, KNIME, Alteryx, and RapidMiner, which I have used extensively.

SAS Viya #

SAS is a powerful analytics and business intelligence tool that offers a range of features for data analysis, reporting, and data visualization. SAS provides a range of tools for building data pipelines, including SAS Viya, which is a visual design tool for building and managing ML data pipelines that can handle a wide range of data sources, transformations, and data quality checks. SAS provides its own proprietary programming language for fine grained control. On the downside, SAS is a very high-end proprietary system that comes with a fairly hefty licensing cost. SAS is required in some highly regulated sectors such as the finance and pharmaceutical industries.

KNIME #

KNIME is an open-source data analytics platform with a free tier, and a paid for enterprise tier. KNIME allows users to build and execute data pipelines for a wide range of data processing and analysis tasks. KNIME provides a visual interface for designing and building data pipelines, and it supports a wide range of data sources and formats. KNIME also provides a range of data processing and machine learning tools, making it a popular choice for data scientists and analysts.

Alteryx #

Alteryx is a data integration and analytics platform that provides a range of tools for data preparation, blending, and analysis. Alteryx provides a visual interface for building data pipelines, and it supports a wide range of data sources and formats. Alteryx also provides a range of pre-built machine learning models that can be used to perform tasks such as regression analysis, classification, and clustering. Alteryx is well-known for its ease of use, and is a good choice for analysts who need to quickly build and deploy models.

RapidMiner #

RapidMiner is an open-source data science platform that allows users to build and execute data pipelines for a wide range of data processing and analysis tasks. RapidMiner provides a visual interface for designing and building data pipelines, and it supports a wide range of data sources and formats. RapidMiner is particularly popular among data scientists and analysts because of its focus on predictive analytics and machine learning.

Conclusion #

Data pipelines are essential for machine learning projects, and there are many tools available for building them that do not require code. All the above tools have the features needed to create complete pipelines, train, evaluate and deploy complete models. The choice of tool can be a matter of preference, so if project costs are an issue, the open source tools with free tiers (KNIME, RapidMiner) are ideal. Alteryx probably has the best ease of use. SAS may be stipulated by industry regulators but in these cases, you’re probably in a large multinational company setting that is well-disposed to swallow the licensing costs.

Data Pipelines and Data Engineers

Thu, 13 May 2021 00:00:00 +0000

Data pipelines are critical for ensuring that data is accurate, timely, and available to those who need it. Without data pipelines, organizations would struggle to process and make sense of the vast amounts of data that they collect. Data pipelines enable organizations to build machine learning models, conduct data analysis, and make data-driven decisions. In this blog post, we will discuss data pipelines and the role of data engineers in building and maintaining them.

What is a data pipeline?

A data pipeline is a series of data processing steps. Data pipelines typically start with extraction from one or more sources. These sources are heteregeneous systems, and sometime public sources like the web and RSS feeds. The data is then sent through cleaning and transformation steps before being loaded into the target system - usually an analytic datastore, data warehouse or data lake. The name Pipelines is apt because it gives us the sense of data flowing from source to sink, passing through different filters and processes on the way. From a technical perspective, pipelines manage large in-memory buffers, where transformative actions are very fast, as long as they don’t depend on any external lookups.

Data pipelines can be either batch or real-time. Batch pipelines process data in batches, which means that data is collected over a period of time, stored in a database, and then processed at once. Real-time pipelines, on the other hand, process data as it is generated or read from real-time logs, and the results are available almost immediately.

The Data Engineer is the data team member responsible for designing, building, and maintaining data pipelines. They are responsible for ensuring that data is collected, stored, and processed in a way that meets the requirements of downstream applications. Data engineers work with a range of tools and technologies to build data pipelines, including databases, data warehouses, ETL (extract, transform, load) tools, and programming languages such as Python, Java, and SQL.

Data engineers must have a strong understanding of data architecture and data modeling, as well as experience working with different types of data sources and data formats. Data engineers often have a computer science background and a deep knowledge of back-end software engineering. They must also be able to design and build data pipelines that can scale to handle large volumes of data.

Conclusion

In conclusion, data pipelines are essential for any organization that wants to make use of the vast amounts of data that it collects. Data engineers are responsible for designing, building, and maintaining data pipelines, and they play a critical role in ensuring that data is accurate, timely, and available to those who need it. Data engineering requires a niche and highly specialised skill set. As the importance of data-driven decision making continues to grow, and data engineers become increasingly important, companies must come up with strategic talent acquisition plans to successfully fill these positions.

Non-Parametric Survival Analysis of a Sleep Diary

Wed, 03 Feb 2021 00:00:00 +0000

Introduction #

Earlier this month I carried out a parametric survival analysis over a self-generated dataset of my sleep times each day over the previous year. Using the scientific method, of course, I set about the task with a null hypothesis that eating certain food groups for dinner had no effect on my sleep. The findings were indeed very interesting, and I was able to reject the null hypothesis using a parametric regression of the data set using a Weibull survival regression. You can read about that here, so I won’t repeat myself and I’ll skip the exploratory analysis. Just remember, the diagnosis is not great! My median nightly sleep time is 5.84 hours, or 05 hours 50 minutes 34 seconds.

In this post, I take the opportunity to explore the same dataset with a more widely used survival analysis, the non-parametric Kaplan-Meyer estimator. As usual, I’ll differ printing out code until the end, unless there is something interesting to show in context.

The K-M Estimator #

The K-M Estimator is calculated cumulatively at each time point where an event occurs.

$$ \hat{S}(t) = \prod_{\substack{t_i \leq t}}{\left( 1 - \frac{d_i}{n_i}\right)} $$ where $n_i$ is the number of observations who have survived up to time $t_i$, or in my case the number of nights where I would still be asleep, and $d_i$ is the number of observations who fail at time $t_i$, or in my case, the number of nights where sleep ends at that elapsed time.

The variance of this estimator is:

$$ \mathrm{var}\left( \hat{S}(t) \right) \approx \left[ {\hat{S}(t)}^2 \right] \sum_{\substack{t_i \leq t}} \frac{d_i}{n_i(n_i - d_i)} $$ and it is usual to take the complementary log-log transform $\mathrm{var} \left( \log \left[ -\log {\hat{S}(t)}^2 \right] \right)$ to constrain the confidence intervals between zero and one.

In a typical survival study, there is an additional factor to consider. Survival studies originate in longitudinal studies of people and conditions generally ending in death. Longitudinal studies are prone to individuals exiting the study over time for other reasons than the events under analysis. When an individual exits the study early, their record is said to be right-censored. It is clipped at the point in time when the individual is no longer observed. It is possible to use the information provided by their length of survival during participation, so long as the uncertainty is also taken into account of not being able to observe if/when the event occurs (e.g. death, relapse, or failure in the case of hardware). Right-censored events increase the variance following the end of their time in the study.

To represent the right-censoring in R, you would provide a Boolean vector of equal length as the observations vector into the Surv object. In my sleep diary, however, each observation is a completely measured night’s sleep and so there aren’t any censored observations. To demonstrate the intricacies of this method, I will first construct the univariate estimator, without adding any co-factors, so the formula is given with just a constant (~1).

1
2

estimator <- survfit(Surv(time_sleeping) ~ 1, conf.type="log-log", data = dframe)
estimator

## Call: survfit(formula = Surv(time_sleeping) ~ 1, data = dframe, conf.type = "log-log")
## 
##        n events median 0.95LCL 0.95UCL
## [1,] 365    365   5.84    5.64     6.1

The median sleeping time and 95% confidence intervals are provided by the estimator based on:

$$ \hat{t}_{\mathrm{med}} = \mathrm{inf} \lbrace t : \hat{S}(t) \leq 0.5 \rbrace $$

R makes everything trivial to calculate. The median sleeping time is 05 hours 50 minutes 34 seconds with a lower confidence estimate of 05 hours 38 minutes 35 seconds and an upper confidence interval of 06 hours 05 minutes 49 seconds.

The red (reference) line represent the level where $\hat{S}(t) = 0.5$ and the green (median) and blue (ci) drop lines show where the reference line intersects with the estimator and its confidence interval.

Typically, survival curves from KM estimators do not look like this. Thanks to the even density of data points (this is essentially a time series), the above plot looks like taking the 365 observations stacked on top of each other in order from the shortest on top to the longest at the bottom. The resulting curve has some similarity to the Weibull curves that I was able to infer using the parametric approach.

Multivariate Analysis #

Categorical Variables #

Non-parametric survival analysis, with it’s roots in clinical trials, is well-developed for comparing two groups and handling perhaps one categorical co-variate with a small number of strata, or one continuous co-variate, but it certainly isn’t common to see the exploratory approach that I used in the previous post on parametric methods.

Sticking to tried and testing methods, I’ll just demonstrate some simple between groups hypothesis testing, using one food trigger at a time. Recall that eating a meal containing cheese for dinner seemed to have a very pronounced effect on me, reducing the median sleep time by about 1 hour 40 minutes when comparing nights when dinner contained none of the analysed food triggers.

I will test the null hypothesis that eating cheese has no effect on sleep times. That is $H_0 : S_C(t) = S_N(t)$ and the alternative is $H_A : S_C(t) \neq S_N(t)$ (or $H_A : S_C(t) < S_N(t)$ for a one-sided test), where $S_C$ is the survival distribution for cheesy nights, and $S_N$ for nights with no food trigger. The standard test is the Mantel-Cox test or log-rank test, which tallies a contingency table for treatment (cheese) and control (no food trigger) observations at each event (end of sleep cycle) time. I won’t reproduce the full derivation here but the resulting statistic follows a $\chi^2$ distribution.

## Call:
## survdiff(formula = Surv(time_sleeping) ~ cheese, data = dframe)
## 
##                N Observed Expected (O-E)^2/E (O-E)^2/V
## cheese=FALSE 301      301    344.5      5.48       103
## cheese=TRUE   64       64     20.5     91.92       103
## 
##  Chisq= 102  on 1 degrees of freedom, p= <2e-16

Evidently, this is a significant result and the null hypothesis is rejected. The effect can be visually analysed with a simple plot.

What’s nice about the KM estimator survival curve is that, unlike a parametric distribution, you can clearly see the location of each event, giving you full transperancy over your empiricial data. In a typical survival study, with perhaps only tens of subjects, the piece-wise nature is really clear and locating sudden changes in the distribution is a matter of a quick visual check.

Let’s take a look at the other food triggers.

## Call:
## survdiff(formula = Surv(time_sleeping) ~ brassica, data = dframe)
## 
##                  N Observed Expected (O-E)^2/E (O-E)^2/V
## brassica=FALSE 244      244    288.3      6.81      33.9
## brassica=TRUE  121      121     76.7     25.59      33.9
## 
##  Chisq= 33.9  on 1 degrees of freedom, p= 6e-09

## Call:
## survdiff(formula = Surv(time_sleeping) ~ meat, data = dframe)
## 
##              N Observed Expected (O-E)^2/E (O-E)^2/V
## meat=FALSE 312      312    329.7     0.953        10
## meat=TRUE   53       53     35.3     8.908        10
## 
##  Chisq= 10  on 1 degrees of freedom, p= 0.002

## Call:
## survdiff(formula = Surv(time_sleeping) ~ spice, data = dframe)
## 
##               N Observed Expected (O-E)^2/E (O-E)^2/V
## spice=FALSE 302      302      335      3.26      41.5
## spice=TRUE   63       63       30     36.43      41.5
## 
##  Chisq= 41.5  on 1 degrees of freedom, p= 1e-10

All the above details are consistent with the parametric findings.

Continuous Variables #

In the previous post, by creating autoregressive features from the time sleeping variable, I identified that the sleep time from the night before affected the sleep time for a given night. I assume that there was some physiological pressure to “catch up.” The same was not true for the previous to last night. We can perform a similar investigation non-parametrically.

I haven’t referred to the hazard function $h(t)$, because I want to keep these posts a bit light on detail, but this is essentially the instantaneous failure rate at time $t$. It is defined as follows:

$$ h(t) = \lim_{\substack{\delta \rightarrow0}} \frac{P(\mathrm{event_T} | t < T < t + \delta | T > t)}{\delta} $$

where $P(\mathrm{event_T})$ is the probability of an event happening at instant T within some tiny time increment $\delta$. There is a mathematical relationship between hazard and survival but it takes a few steps to derive it, so I’ll be a bit hand-wavey here and just say that where $h(t)$ is high, $S(t)$ is falling fast.

Analysis of hazard functions is done by Cox’s Proportional Hazards, which uses a log-likelihood statistic and can be used for survival regression. The coefficients of a proportional hazards regression analysis follow a normal distribution and so can be subject to familiar significance tests.

Let’s quickly see how this looks for the cheese trigger to develop some intuition before using it to assess the autoregressive effect of previous night sleeping time.

## Call:
## coxph(formula = Surv(time_sleeping) ~ factor(cheese), data = dframe)
## 
##   n= 365, number of events= 365 
## 
##                      coef exp(coef) se(coef)     z Pr(>|z|)    
## factor(cheese)TRUE 1.4012    4.0603   0.1492 9.391   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
##                    exp(coef) exp(-coef) lower .95 upper .95
## factor(cheese)TRUE      4.06     0.2463     3.031      5.44
## 
## Concordance= 0.591  (se = 0.011 )
## Likelihood ratio test= 69.73  on 1 df,   p=<2e-16
## Wald test            = 88.19  on 1 df,   p=<2e-16
## Score (logrank) test = 102.5  on 1 df,   p=<2e-16

This analysis is indicating a log proportional hazard of 1.4012452 for nights with a meal containing cheese (zero for no cheese), which is statistically significant.

Now for the analysis for my autoregressive features of one and two nights previous. I don’t anticipate a linear relationship so I will pass a penalized spline into the model.

## Call:
## coxph(formula = Surv(time_sleeping) ~ pspline(ar_1, df = 2) + 
##     pspline(ar_2, df = 2), data = dframe)
## 
##                              coef se(coef)     se2   Chisq   DF      p
## pspline(ar_1, df = 2), li -0.1021   0.0334  0.0333  9.3549 1.00 0.0022
## pspline(ar_1, df = 2), no                           2.1989 1.06 0.1494
## pspline(ar_2, df = 2), li -0.1691   0.0343  0.0342 24.3589 1.00  8e-07
## pspline(ar_2, df = 2), no                           2.4917 1.05 0.1224
## 
## Iterations: 4 outer, 13 Newton-Raphson
##      Theta= 0.936 
##      Theta= 0.925 
## Degrees of freedom for terms= 2.1 2.1 
## Likelihood ratio test=41.4  on 4.12 df, p=3e-08
## n= 365, number of events= 365

This is an interesting result that differs from the parametric findings. The print out shows a coefficient for each spline’s linear part, and a significance test only for the non-linear part. Sleep time on both the previous night (coef = -0.1021) and two nights previous (coef = -0.1691) have an effect, with two nights previous being slightly larger. Both coefficients are found to be significant and negative, and so inversely proportional. The interpretation is that less time sleeping on previous nights increases the harzard. This runs counter to intuition and the parametric findings. However, R provides a built in plotting function that reveals the relationship very clearly.

The plots confirm the model results that hazard increases with less sleep on the previous two nights. There is an inflection point somewhere near the median sleep time where the effect is neutral in both cases. The non-intuitive inverse relationship could be a result of not running the other co-factors in the model (something akin to Simpson’s paradox).

Unlike the KM estimator survival curve analysis, it’s more intuitive to run a multivariate proportional hazards regression with many more co-factors. I’ll run the model with the seasonal (month) effect included to see if my hunch is correct.

## Call:
## coxph(formula = Surv(time_sleeping) ~ pspline(ar_1, df = 2) + 
##     pspline(ar_2, df = 2) + month, data = dframe)
## 
##                              coef se(coef)     se2   Chisq   DF       p
## pspline(ar_1, df = 2), li  0.1051   0.0418  0.0418  6.3128 1.00 0.01199
## pspline(ar_1, df = 2), no                           1.7641 1.05 0.19541
## pspline(ar_2, df = 2), li -0.0078   0.0411  0.0411  0.0360 1.00 0.84960
## pspline(ar_2, df = 2), no                           4.7869 1.05 0.03084
## monthAug                   1.6808   0.2938  0.2932 32.7242 1.00 1.1e-08
## monthDec                  -0.5794   0.2685  0.2673  4.6548 1.00 0.03097
## monthFeb                  -0.7621   0.2776  0.2767  7.5353 1.00 0.00605
## monthJan                  -0.9732   0.2801  0.2784 12.0708 1.00 0.00051
## monthJul                   1.8935   0.2978  0.2967 40.4382 1.00 2.0e-10
## monthJun                   0.6349   0.2656  0.2651  5.7124 1.00 0.01685
## monthMar                  -0.4502   0.2631  0.2628  2.9282 1.00 0.08704
## monthMay                   0.2949   0.2612  0.2605  1.2747 1.00 0.25889
## monthNov                  -0.3267   0.2609  0.2607  1.5679 1.00 0.21051
## monthOct                  -0.0930   0.2586  0.2583  0.1292 1.00 0.71923
## monthSep                   0.6727   0.2655  0.2652  6.4219 1.00 0.01127
## 
## Iterations: 4 outer, 13 Newton-Raphson
##      Theta= 0.924 
##      Theta= 0.918 
## Degrees of freedom for terms=  2.1  2.0 10.9 
## Likelihood ratio test=152  on 15 df, p=<2e-16
## n= 365, number of events= 365

Wow! I was absolutely spot on. Controlling for the seasonal effect of just sleeping less in the shorter summer nights, the results are consistent with the findings from last time. Sleep time on the previous night (coef = 0.1051) has a significant effect that is no longer reversed, and two nights previous (coef = -0.0078) no longer has a significant effect.

The plots confirm this finding, with previous night hazard appearing to fall monotonically with less sleep, while for two nights previous, the zero reference line is entirely contained inside the confidence interval.

Further work #

It is possible to go further with proportional hazards regression and to select the best model using log-likelihood tests for nested models and AIC for non-nested models. This is familiar territory for linear models and GLMs. I performed this procedure in the previous post on parametric survival analysis. So for the moment, my investigation will end here.

Conclusion #

Running a typical survival analysis on an atypical dataset was an interesting exercise. It did yield a confusing result as I progressed to a more advanced regression analysis with autoregressive continuous variables at first, but this was cleared up by approaching the problem with a critical mindset.

The parametric approach from the previous post yielded a very satisfying analysis because the data was well-fitting to the Weibull distribution, as well as acheiving goodness of fit with a Gamma distribution and normal distribution. However, the non-parametric survival curves yielded by the Kaplan-Meyer estimator actually show you the true, empirical picture of your data set rather than some theoretical distribution. In most cases, this is preferable to work with.

We couldn’t look into some of the aspects of survival analysis that only present themselves in longitudinal studies, such as observations being censored by exiting the study prior to the analysed events. I may return to this topic in a future post.

Overall, it was really great fun to work with a self-generated data set that defines a problem that is pretty core to my personal life. I learned a lot about how to manage my chronic insomnia and that can only be a good thing.

Code Appendix #

library(lattice)
library(ggplot2)
library(dplyr)
library(tidyr)
library(tibble)
library(survival)
library(Hmisc)
library(goftest)
library(fitdistrplus)
library(patchwork)

`1`	`dframe <- read.csv("data/insomnia-diary.csv")`

days_in_months <- c(31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31)
month_vector <- rep(1:12, times = days_in_months)
dframe$month_number <- month_vector
month_name <- c("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
month_positions <-c(0, cumsum(days_in_months) + 1)[-13]
dframe$month <- month_name[dframe$month_number]

dframe$quarter <- (dframe$month_number - 1) %/% 3 + 1
dframe[["combo_foods"]] <- factor(with(dframe, cheese * 1 + brassica * 2 + meat * 4 + spice * 8))

dframe[["trend"]] = 1:length(dframe$time_sleeping)
dframe[["ar_1"]] = c(dframe$time_sleeping[1], head(dframe$time_sleeping, -1))
dframe[["ar_2"]] = c(rep(dframe$time_sleeping[1], 2), head(dframe$time_sleeping, -2))
dframe[["ar_3"]] = c(rep(dframe$time_sleeping[1], 3), head(dframe$time_sleeping, -3))
dframe[["ar_4"]] = c(rep(dframe$time_sleeping[1], 4), head(dframe$time_sleeping, -4))

convert_hms <- function(hours) {
  total_seconds <- hours * 3600
  hrs <- as.integer(floor(total_seconds / 3600))
  mins <- as.integer(floor((total_seconds %% 3600) / 60))
  secs <- as.integer(total_seconds %% 60)

  sprintf("%02d hours %02d minutes %02d seconds", hrs, mins, secs)
}

1
2

estimator <- survfit(Surv(time_sleeping) ~ 1, conf.type="log-log", data = dframe)
estimator

med <- quantile(estimator, 0.5)
med_sleep <- med$`quantile`
lc_sleep <- med$lower
uc_sleep <- med$upper

plot(estimator, conf.int = TRUE, xlab="Time (hr)", ylab = "Survival (still sleeping) Probability")
title("Kaplan-Meyer Estimator of Insomnia Diary Data")
abline(h = 0.5, col = "red", lty = 3)
lines(rep(x = uc_sleep, 2), y = c(0, 0.5), col = "blue", lty = 2)
lines(rep(x = lc_sleep, 2), y = c(0, 0.5), col = "blue", lty = 2)
lines(rep(x = med_sleep, 2), y = c(0, 0.5), col = "green", lty = 2)

`1`	`survdiff(Surv(time_sleeping) ~ cheese, data = dframe)`

1
2

plot(survfit(Surv(time_sleeping) ~ cheese, data = dframe), xlab="Time (hr)", ylab = "Survival (still sleeping) Probability", col=c("black", "red"), lwd = 1)
legend("topright", legend=c("no cheese", "cheese"), col=c("black", "red"), lwd = 1)

`1`	`survdiff(Surv(time_sleeping) ~ brassica, data = dframe)`

1
2

plot(survfit(Surv(time_sleeping) ~ brassica, data = dframe), xlab="Time (hr)", ylab = "Survival (still sleeping) Probability", col=c("black", "red"), lwd = 1)
legend("topright", legend=c("no brassica", "brassica"), col=c("black", "red"), lwd = 1)

`1`	`survdiff(Surv(time_sleeping) ~ meat, data = dframe)`

1
2

plot(survfit(Surv(time_sleeping) ~ meat, data = dframe), xlab="Time (hr)", ylab = "Survival (still sleeping) Probability", col=c("black", "red"), lwd = 1)
legend("topright", legend=c("no meat", "meat"), col=c("black", "red"), lwd = 1)

`1`	`survdiff(Surv(time_sleeping) ~ spice, data = dframe)`

1
2

plot(survfit(Surv(time_sleeping) ~ spice, data = dframe), xlab="Time (hr)", ylab = "Survival (still sleeping) Probability", col=c("black", "red"), lwd = 1)
legend("topright", legend=c("no spice", "spice"), col=c("black", "red"), lwd = 1)

1
2

cph_cheese <- coxph(Surv(time_sleeping) ~ factor(cheese), data = dframe)
summary(cph_cheese)

`1`	`termplot(cph_cheese, se = TRUE, terms = 1, ylabs = "Log hazard")`

1
2

cph_ar <- coxph(Surv(time_sleeping) ~ pspline(ar_1, df=2) + pspline(ar_2, df=2), data=dframe)
cph_ar

termplot(cph_ar, se = TRUE, terms = 1, ylabs = "Log hazard", xlabs = "Previous Night")
abline(h = 0, col="grey", lty=3)

termplot(cph_ar, se = TRUE, terms = 2, ylabs = "Log hazard", xlabs = "Two Nights Previous")
abline(h = 0, col="grey", lty=3)

1
2

cph_ar <- coxph(Surv(time_sleeping) ~ pspline(ar_1, df=2) + pspline(ar_2, df=2) + month, data=dframe)
cph_ar

1
2

summary_output <- summary(cph_ar)
linear_coeffs <- summary_output$coefficients

termplot(cph_ar, se = TRUE, terms = 1, ylabs = "Log hazard", xlabs = "Previous Night")
abline(h = 0, col="grey", lty=3)

termplot(cph_ar, se = TRUE, terms = 2, ylabs = "Log hazard", xlabs = "Two Nights Previous")
abline(h = 0, col="grey", lty=3)

Parametric Survival Analysis of a Sleep Diary

Fri, 15 Jan 2021 00:00:00 +0000

Introduction #

As a teenager, I began to suffer with chronic and ongoing insomnia that has stayed with me for all of my adult life with very few periods of real respite. However, rather than turn this into a sob story, I’ve managed to come up with a really interesting data story!

In the search for triggers and patterns, I decided to buy a wearable fitness tracker and keep a tally of my nightly quality sleep hours. Around the same time, I finally had the self-awareness to realise that a lot of my worst nights seemed to be accompanied by digestive disturbances, even so to hypothesise that certain food groups may be acting as triggers or exacerbating the problem. I decided to keep a food diary alongside the sleep data. I might also add that I’m proud of myself for not skipping a day during the collection period.

This post will detail some of the most interesting findings after one year of this fascinating self-study. I want to demonstrate a case for using survival analysis for this dataset, because I’m analysing a time to event (waking up), which is made possible by using the wearable device that tracks quality sleep hours and is agnostic of what time I go to bed or if I wake up in the middle of the night.

As usual with these posts, I don’t print the code inline so we can see the analysis step-by-step, which is more interesting for most people. All the code is then echoed in the appendix.

Exploratory #

The raw dataset consists of five variables and 365 observations. The observations are ordered (but not labelled) by date, ranging from 2022-01-01 to 2022-12-31. The second is the dependent variable: time_sleeping, which for simplicity is measured in hrs (float), rather than calculating out minutes and seconds. The remaining four are boolean flags for whether a food group was present or absent in that evening’s meal. The food groups were based on suggestions by my GP and personal observation. Namely, cheese, brassica, (red) meat, and spice (including chili, curry, onions and garlic).

1
2

##    vars   n mean   sd median trimmed  mad  min   max range skew kurtosis   se
## X1    1 365 5.97 1.61   5.84    5.96 1.63 1.62 10.72   9.1 0.09    -0.24 0.08

## dplyr::select(dframe, cheese, brassica, meat, spice) 
## 
##  4  Variables      365  Observations
## --------------------------------------------------------------------------------
## cheese 
##        n  missing distinct 
##      365        0        2 
##                       
## Value      FALSE  TRUE
## Frequency    301    64
## Proportion 0.825 0.175
## --------------------------------------------------------------------------------
## brassica 
##        n  missing distinct 
##      365        0        2 
##                       
## Value      FALSE  TRUE
## Frequency    244   121
## Proportion 0.668 0.332
## --------------------------------------------------------------------------------
## meat 
##        n  missing distinct 
##      365        0        2 
##                       
## Value      FALSE  TRUE
## Frequency    312    53
## Proportion 0.855 0.145
## --------------------------------------------------------------------------------
## spice 
##        n  missing distinct 
##      365        0        2 
##                       
## Value      FALSE  TRUE
## Frequency    302    63
## Proportion 0.827 0.173
## --------------------------------------------------------------------------------

The diagnosis is not great! My median nightly sleep time is 5.84 hours, or 05 hours 50 minutes 34 seconds.

It’s also clear that I enjoy spicy food on occasion, and I notice the seasonal pattern, sleeping much longer in winter than in summer.

Hypothesis And Analytical Method #

The null hypothesis that I can test with these data is that none of these food groups have any effect on my sleep. The alternative is that at least one has an effect. And that’s exactly what I want to know if I am to improve my sleep by changing my eating habits. Hopefully, this knowledge will allow me to continue eating most of the things I enjoy.

Gotchas #

A multi-variate linear model would normally be a starting point but there are several potential confounders:

Some evenings, dinner can contain one, two or even three of these food groups. The effects may be hard to isolate.
There is likely to be some auto-regressive influence, for example, two or three nights of poor sleep are sometimes followed by a catch-up night where the mind and body are just exhausted and sleep for longer.
Sleep is highly seasonal, being definitely shorter in summer as the hours of darkness are shorter in Germany, where I live.

Other Considerations #

Time sleeping is the total time that the fitness tracker detected quality sleep. However, this is a somewhat simplified view because my sleep patterns tend to be broken and interrupted. I go to bed and get up and approximately the same time most days but some nights I have to deal with two to three hours of wakeful restlessness. For simplicity, I’m counting this all as one sleep event of a certain length of time. It was too much trouble to count or measure the sleep interruptions. The resulting data are always a positive real number. That is, there are no nights of negative sleep and any model should reflect this. A linear model, in theory, could return a negative value, so it is necessary to consider some alternatives.

Time-to-event is often modeled using Gamma or Weibull distributions, and well understood within Survival Analytics. So, to test this line of thinking, I would like to compare three models: Linear, Generalized Linear with Gamma family, and a parametric survival model with Weibull distribution.

Pre-requisites #

I use the versatile Anderson-Darling test for goodness of fit with the three candidate distributions. Although other tests such as Kolmogorov-Smirnov and Shapiro-Wilks are most common for goodness of fit to the Normal distribution, Anderson-Darling still works well for the Normal, and given the decent sample size and putting aside the likely presence of auto-correlation, it’s very useful to be able to use the same test for all three candidates. Anderson-Darling test is most sensitive to deviations in the tail, and so is an excellent choice for making assessments of fit to both Gamma and Weibull distributions.

One disadvantage is that the tests assume fixed, known parameters. However, these are unknown and have to be estimated directly from the data to provide to the test routines. This is not recommended practice and can lead to biased results. On this occasion, I will accept the risk because it is outweighed by the convenience of assessing all three fits with the same test. I can apply a further visual check with a Q-Q plot should be enough to confirm or reject the goodness-of-fit test results.

## 
## 	Anderson-Darling test of goodness-of-fit
## 	Null hypothesis: Normal distribution
## 	with parameters mean = 5.97381266124959, sd = 1.61023592966754
## 	Parameters assumed to be fixed
## 
## data:  dframe$time_sleeping
## An = 0.41908, p-value = 0.8295

## 
## 	Anderson-Darling test of goodness-of-fit
## 	Null hypothesis: Gamma distribution
## 	with parameters shape = 12.6312583519882, rate = 2.11434807107313
## 	Parameters assumed to be fixed
## 
## data:  dframe$time_sleeping
## An = 1.0658, p-value = 0.3245

## 
## 	Anderson-Darling test of goodness-of-fit
## 	Null hypothesis: Weibull distribution
## 	with parameters shape = 4.09387276140609, scale = 6.5788628337449
## 	Parameters assumed to be fixed
## 
## data:  dframe$time_sleeping
## An = 0.60368, p-value = 0.6444

It’s worth comparing these theoretical distributions with the sample to understand why all three tests suggests a goodness of fit. Clearly there are pros and cons to using any of these distributions to model the data. The Gamma density curve is better aligned with the sample peak but the over-estimated skew is evident. It could be biased to under-estimate extreme values. Without calculating the KL-divergence, the Weibull has the visual appearance of balancing out the mass around the peak better than the normal but looks like it will be biased to over-estimating the mean. However, it is guaranteed never to return a negative real value, unlike the Normal. The Normal distribution is still a good fit, and comes with the richest literature and support. I will proceed with the three-way comparison.

Comparing Models #

To mitigate the possibility of auto-regressive, trend and combined food-trigger confounders, I add the following features:

auto-regressive features (up to four days of lag)
testing with daily index vs. month number vs. calendar quarter for isolating the seasonal trend
combine the four Booleans variables bitwise to create a factor variable with a unique value for each combination of one, two and three food triggers

For each model family, I tested different model formulas, trying either trend (linear unit increase per day), month (number), month (factor), quarter (number), and quarter (factor) for the time based component, with and without quadratic components for numeric time-based components. I tried with the four separate Boolean food triggers, and separately with the unique combination food triggers (factor). When the best performing model was identified in each family, I removed non-significant variables among the four auto-regressive components.

In all three cases, there was complete alignment on the parameter set that yielded the best model. The month (factor) was sufficient to capture differences in sleeping time attributable to changes in daylight hours, while keeping the food triggers as separate Booleans was better for explaining the variance than combining them into a single, multi-level factor. This is likely because the combined factor almost doubles the number of levels.

Finally, it was clear that only the one day lag was needed to account for catchup sleep after a previous bad night.

I’ll skip the model selection and just report the best performing model from each family.

## 
## Call:
## lm(formula = time_sleeping ~ ., data = model_frame)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.9755 -0.5928  0.0967  0.6734  3.1957 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   7.44056    0.30546  24.358  < 2e-16 ***
## ar_1         -0.08339    0.03934  -2.120  0.03473 *  
## monthAug     -1.57179    0.26805  -5.864 1.05e-08 ***
## monthDec      0.53566    0.26206   2.044  0.04170 *  
## monthFeb      1.13004    0.27302   4.139 4.38e-05 ***
## monthJan      1.32557    0.26444   5.013 8.56e-07 ***
## monthJul     -1.79690    0.26979  -6.660 1.07e-10 ***
## monthJun     -0.66480    0.26355  -2.522  0.01210 *  
## monthMar      0.64005    0.26324   2.431  0.01554 *  
## monthMay     -0.36251    0.26289  -1.379  0.16879    
## monthNov      0.45920    0.26401   1.739  0.08286 .  
## monthOct      0.03702    0.26169   0.141  0.88759    
## monthSep     -0.71104    0.26485  -2.685  0.00761 ** 
## cheeseTRUE   -1.67430    0.14252 -11.748  < 2e-16 ***
## brassicaTRUE -1.01451    0.11468  -8.846  < 2e-16 ***
## meatTRUE     -0.46953    0.15414  -3.046  0.00249 ** 
## spiceTRUE    -1.04497    0.14284  -7.316 1.78e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.017 on 348 degrees of freedom
## Multiple R-squared:  0.6183,	Adjusted R-squared:  0.6008 
## F-statistic: 35.24 on 16 and 348 DF,  p-value: < 2.2e-16

`1`	`## [1] AIC: 1067.007`

## 
## Call:
## glm(formula = time_sleeping ~ ., family = Gamma, data = model_frame)
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   0.1311168  0.0082441  15.904  < 2e-16 ***
## ar_1          0.0017776  0.0010769   1.651 0.099703 .  
## monthAug      0.0546879  0.0086455   6.326 7.75e-10 ***
## monthDec     -0.0130204  0.0071211  -1.828 0.068341 .  
## monthFeb     -0.0242873  0.0070560  -3.442 0.000648 ***
## monthJan     -0.0293840  0.0067765  -4.336 1.90e-05 ***
## monthJul      0.0679070  0.0090650   7.491 5.69e-13 ***
## monthJun      0.0186200  0.0075813   2.456 0.014535 *  
## monthMar     -0.0147436  0.0070126  -2.102 0.036232 *  
## monthMay      0.0115290  0.0074912   1.539 0.124711    
## monthNov     -0.0096972  0.0070557  -1.374 0.170211    
## monthOct     -0.0001734  0.0071671  -0.024 0.980714    
## monthSep      0.0204151  0.0076836   2.657 0.008248 ** 
## cheeseTRUE    0.0579108  0.0050391  11.492  < 2e-16 ***
## brassicaTRUE  0.0287559  0.0034145   8.422 9.86e-16 ***
## meatTRUE      0.0124766  0.0047046   2.652 0.008367 ** 
## spiceTRUE     0.0319680  0.0045929   6.960 1.70e-11 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for Gamma family taken to be 0.02992814)
## 
##     Null deviance: 29.262  on 364  degrees of freedom
## Residual deviance: 11.639  on 348  degrees of freedom
## AIC: 1091.7
## 
## Number of Fisher Scoring iterations: 4

`1`	`## [1] AIC: 1091.679`

## 
## Call:
## survreg(formula = Surv(time_sleeping) ~ ., data = model_frame)
##                 Value Std. Error      z       p
## (Intercept)   2.06657    0.04542  45.50 < 2e-16
## ar_1         -0.01259    0.00602  -2.09 0.03644
## monthAug     -0.30578    0.03881  -7.88 3.3e-15
## monthDec      0.13000    0.03747   3.47 0.00052
## monthFeb      0.14466    0.03880   3.73 0.00019
## monthJan      0.20162    0.03799   5.31 1.1e-07
## monthJul     -0.31096    0.03957  -7.86 3.9e-15
## monthJun     -0.11504    0.03722  -3.09 0.00200
## monthMar      0.07461    0.03740   2.00 0.04604
## monthMay     -0.07236    0.03778  -1.92 0.05547
## monthNov      0.07050    0.03797   1.86 0.06337
## monthOct     -0.00262    0.03731  -0.07 0.94398
## monthSep     -0.11409    0.03808  -3.00 0.00274
## cheeseTRUE   -0.28019    0.02082 -13.46 < 2e-16
## brassicaTRUE -0.12376    0.01677  -7.38 1.6e-13
## meatTRUE     -0.07969    0.02241  -3.56 0.00038
## spiceTRUE    -0.19399    0.02057  -9.43 < 2e-16
## Log(scale)   -1.93934    0.04068 -47.68 < 2e-16
## 
## Scale= 0.144 
## 
## Weibull distribution
## Loglik(model)= -504.6   Loglik(intercept only)= -692.3
## 	Chisq= 375.37 on 16 degrees of freedom, p= 5.2e-70 
## Number of Newton-Raphson Iterations: 6 
## n= 365

`1`	`## [1] AIC: 1045.296`

Diagnostics #

Based on the AIC results, it appears as though the parametric survival model with Weibull distribution is the best of the three models, and the Generalized Linear Model with Gamma distribition and inverse link function is the least well performing. I will conduct a likelihood ratio test to see if the differences between the three models are significant.

`1`	`## [1] WB > LM, Likelihood Ratio Test Statistic: 21.711, p-value: 0.000`

`1`	`## [1] WB > GLM, Likelihood Ratio Test Statistic: 46.382, p-value: 0.000`

`1`	`## [1] LM > GLM, Likelihood Ratio Test Statistic: 24.671, p-value: 0.000`

These results suggest that the parametric survival model with Weibull distribution is the best model over all by a significant margin.

I will show diagnostic plots for this model only but none of the models showed any cause for concern. Note, for a Weibull model, we expect the deviance residuals to be i.i.d.

Results and Interpretation #

Coefficients #

It is easiest to start with the linear model because the coefficients are easiest to interpret. Aside from the auto-regressive component, each coefficient represents the value in hours to add (or subtract) from the Intercept of 7.44 hours or 07 hours 26 minutes 26 seconds. For example, I should expect my mean sleeping hours to be 7.44 -1.8 during the month of July, or 05 hours 38 minutes 37 seconds (ignoring the standard errors, for simplicity). Similarly, eating cheese for dinner has a mean effect of -1.67. This was the strongest of the four food triggers.

Good job I never eat fondue in the summer!

The Generalized linear model has a slightly different interpretation. We would calculate the linear predictor from the coefficients just the same as with the linear model, but then take the inverse to get the prediction response. As a result, the coefficients have the opposite sign as that given by the linear model coefficients, and the magnitudes are less intuitive to quantify. The base case of month April (alphabetically the first, so set as the reference level by R) and no food triggers, the intercept is $\frac{1}{0.1311168}$ or 07 hours 26 minutes 26 seconds (exactly the same value as the linear model, which is expected). This predictor grows or shrinks linearly with the coefficients, but multiplicatively on the response scale. This is the equivalent of growing or shrinking the intercept as a starting value by a factor of $\frac{\beta_0}{\beta_0 + \beta_1 + \beta_2 \ldots}$ where $\beta_0$ is the Intercept and $\beta_1 + \beta_2 \ldots$ are the remaining coefficients whose criteria evaluate as true.

In both of the above descriptions, the auto-regressive component adds a multiple of its coefficient value equal to the time sleeping of the day before.

The Weibull model’s coefficients are also used to generate a linear predictor but the relationship of this predictor is even less intuitive than the inverse linked Gamma model. The link function is

$$ \hat{y} = e^{\eta} \times \Gamma \big(1 + \frac{1}{k}\big) $$

where $\eta = X\beta$, is the linear predictor and $k$ is the Weibull shape parameter, and it’s log inverse $\frac{1}{k}$ is returned as Log(scale) in the model summary. We can see from the above that the coefficients are also on the log scale of the prediction response. However, usefully (and unlike the Gamma model), their sign is aligned with their increasing or decreasing effect on the time sleeping response. So, for the base case (month of April and no food triggers), the linear predictor is simply 2.0665712, which exponentiated give us 7.8976967. Log($k$) from the model summary is 1.9393441 and so $k$ is 6.9541879 and it’s inverse is 0.1437982, which will plug into the equation above. The predicted response is therefore:

$$ \hat{y} = e^{2.0665712} \times \Gamma \big(1 + 0.1437982\big) = 7.8976967 \times 0.9351178 = 7.3852764 $$

which at 07 hours 23 minutes 06 seconds is a little shorter than base case for the linear model and generalized linear model described above.

We can use the base case of month April and no food triggers to investigate the auto-regressive effect quality of sleep the night before.

## Night before 06 hours 53 minutes 51 seconds -> predicted time 07 hours 14 minutes 26 seconds
## Night before 07 hours 23 minutes 51 seconds -> predicted time 07 hours 11 minutes 43 seconds
## Night before 07 hours 53 minutes 51 seconds -> predicted time 07 hours 09 minutes 00 seconds
## Night before 08 hours 23 minutes 51 seconds -> predicted time 07 hours 06 minutes 19 seconds
## Night before 08 hours 53 minutes 51 seconds -> predicted time 07 hours 03 minutes 38 seconds

We can see that around the base case, $\pm$ each 30 minutes sleep leads to an inverse (change of sign) change of $\pm \approx$ 3 minutes. Recall that the linear predictor is on the log scale, so the effect is not linear with the previous night’s sleeping time.

Fitted vs. Actual #

Here I show a visual comparison of the true and predicted values from each model. Not too much to say, except the linear model seems to be biased somewhat toward under-estimating sleeping time.

The Survival Analytics Approach #

So far, I’ve diagnosed and intepreted the Weibull model as if it is a linear model, but survival analysis deals with the estimation in terms of a survival function, the probability of surviving (sleeping) up to a point in time $t$:

$$ S(t) = P(T > t),\ 0 < t < \infty $$

where, in this case $T$ is time sleeping. For $t = 0$, $S(t) = 1$ and is monotonically decreasing with increasing $t$. The survival function for a Weibull model is:

$$ S(t) = e^{\big(-(\frac{t}{\lambda})^\mathrm{shape} \big)} $$

Point predictions from such a model are equal to the expected value, of course. However, the Weibull distribution is very flexible, and this point prediction may be far from the most common outcome. On the other hand, the survival curve is useful because it is so easy to estimate the median or a quartile range, for example. This is much easier to understand. On any night with the same conditions, I will sleep this long or shorter/longer 50% of the time. To see the utility of this, we can plot survival curves for various combinations of food triggers at different times of year, and we have an intuitive interpretation as the diminishing chance to still be asleep after a certain length of time has elapsed.

First, I will review the base case $\pm$ 30 minutes and 1 hour.

## Night before 06 hours 53 minutes 51 seconds -> predicted median 06 hours 52 minutes 08 seconds
## Night before 07 hours 23 minutes 51 seconds -> predicted median 06 hours 49 minutes 33 seconds
## Night before 07 hours 53 minutes 51 seconds -> predicted median 06 hours 46 minutes 59 seconds
## Night before 08 hours 23 minutes 51 seconds -> predicted median 06 hours 44 minutes 25 seconds
## Night before 08 hours 53 minutes 51 seconds -> predicted median 06 hours 41 minutes 53 seconds

Then we can try the base case with different food triggers:

## Food trigger none -> predicted median 06 hours 46 minutes 59 seconds
## Food trigger cheese -> predicted median 05 hours 07 minutes 32 seconds
## Food trigger brassica -> predicted median 05 hours 59 minutes 36 seconds
## Food trigger meat -> predicted median 06 hours 15 minutes 48 seconds
## Food trigger spice -> predicted median 05 hours 35 minutes 13 seconds

I am quite shocked at just how pronounced the effect is of these different foods are on my sleep patterns. I have also ascertained that the effect is linear on the log time scale, which I interpret to imply that the effect of combining these foods is diminishing, relative to the hit of taking any one of them for dinner. Cheese is especially bad, which is not suprising as I have a medical lactose intolerance. However, it’s taken until now to really understand the impact that this has had on my quality of life.

As a final analysis, let’s take a look at seasonal effects, controlling for everything else:

## Month Jan -> predicted median 08 hours 17 minutes 53 seconds
## Month Feb -> predicted median 07 hours 50 minutes 19 seconds
## Month Mar -> predicted median 07 hours 18 minutes 30 seconds
## Month Apr -> predicted median 06 hours 46 minutes 59 seconds
## Month May -> predicted median 06 hours 18 minutes 34 seconds
## Month Jun -> predicted median 06 hours 02 minutes 45 seconds
## Month Jul -> predicted median 04 hours 58 minutes 12 seconds
## Month Aug -> predicted median 04 hours 59 minutes 45 seconds
## Month Sep -> predicted median 06 hours 03 minutes 06 seconds
## Month Oct -> predicted median 06 hours 45 minutes 55 seconds
## Month Nov -> predicted median 07 hours 16 minutes 42 seconds
## Month Dec -> predicted median 07 hours 43 minutes 29 seconds

Here again, it’s a shock to see just how bad things get during the summer months when the hours of darkness are especially short. I should surely invest in a sleep mask.

Summary #

Having suffered with insomnia my whole life and sometimes been at a loss to know what to do about it, I decided to create a personal dataset with a fitness tracker and a food diary.

I compared and contrasted three regression models, and selected a Weibull regression as the best model. This model had the following characteristics:

The lowest AIC score of the three models.
Was significantly better than the other two models in a likelihood ratio test
Supports regressing on an auto-regressive component, significant because sleep quality the day before affects the following day
Guarantees never to predict a negative number (impossible for this real-world applied scenario)
Has an intuitive interpretation, using survival analysis

I then demonstrated the difference between the point prediction and the survival curve. The survival curve allows me to estimate the median time I would expect to sleep, controlling for different conditions. This is by far the most intuitive reading of the predictor because the linear predictor is on the log time scale and is further transformed with a gamma function term to be arithmetically correct, rendering the relationship between the coefficients and the prediction harder to reason about.

Code Appendix #

days_in_months <- c(31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31)
month_vector <- rep(1:12, times = days_in_months)
dframe$month_number <- month_vector
month_name <- c("Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
month_positions <-c(0, cumsum(days_in_months) + 1)[-13]
dframe$month <- month_name[dframe$month_number]

ad_test_normal <- ad.test(dframe$time_sleeping, pnorm, mean = mean(dframe$time_sleeping), sd = sd(dframe$time_sleeping))

print(ad_test_normal)

qqnorm(dframe$time_sleeping, main = "Q-Q Plot for Normal Distribution")
qqline(dframe$time_sleeping, col = "red")

# estimate the shape parameter
fit_gamma <- fitdist(dframe$time_sleeping, "gamma")

ad_test_gamma <- ad.test(
  dframe$time_sleeping,
  pgamma,
  shape = fit_gamma$estimate["shape"],
  rate = fit_gamma$estimate["rate"]
  )
print(ad_test_gamma)

qqplot(
  qgamma(
    ppoints(dframe$time_sleeping),
    shape = fit_gamma$estimate["shape"],
    rate = fit_gamma$estimate["rate"]),
  dframe$time_sleeping,
  main = "Q-Q Plot for Gamma Distribution",
  xlab = "Theoretical Quantiles",
  ylab = "Sample Quantiles"
  )
abline(0, 1, col = "red")

fit_weibull <- fitdist(dframe$time_sleeping, "weibull")

ad_test_weibull <- ad.test(
  dframe$time_sleeping,
  pweibull,
  shape = fit_weibull$estimate["shape"],
  scale = fit_weibull$estimate["scale"]
  )
print(ad_test_weibull)


qqplot(
  qweibull(
    ppoints(dframe$time_sleeping),
    shape = fit_weibull$estimate["shape"],
    scale = fit_weibull$estimate["scale"]
    ),
  dframe$time_sleeping,
  main = "Q-Q Plot for Weibull Distribution",
  xlab = "Theoretical Quantiles",
  ylab = "Sample Quantiles")
abline(0, 1, col="red")

ggplot(dframe, aes(x = time_sleeping)) +
  geom_density(fill = "steelblue", color = "black", alpha = 0.7) +  
  stat_function(fun = dnorm, args = list(mean = mean(dframe$time_sleeping), sd = sd(dframe$time_sleeping)), 
                aes(color = "Normal Distribution", linetype = "Normal Distribution"), size = 0.8) +
  stat_function(fun = dgamma, args = list(shape = fit_gamma$estimate["shape"], rate = fit_gamma$estimate["rate"]), 
                aes(color = "Gamma Distribution", linetype = "Gamma Distribution"), size = 0.8) +
  stat_function(fun = dweibull, args = list(shape = fit_weibull$estimate["shape"], scale = fit_weibull$estimate["scale"]), 
                aes(color = "Weibull Distribution", linetype = "Weibull Distribution"), size = 0.8) +
  labs(title = "Density Plot of Time Sleeping with Theoretical Distributions", 
       x = "Time Sleeping (hours)", 
       y = "Density", 
       color = "Distributions", 
       linetype = "Distributions") +
  scale_color_manual(values = c("Normal Distribution" = "#CC503E",    
                                "Gamma Distribution" = "#2E8B57",    
                                "Weibull Distribution" = "#6A0DAD")) +
  scale_linetype_manual(values = c("Normal Distribution" = "solid",    
                                   "Gamma Distribution" = "dashed",    
                                   "Weibull Distribution" = "dotdash")) +
  theme_minimal() +
  theme(legend.position = "top", 
        legend.key.width = unit(3, "picas"))

dframe$quarter <- (dframe$month_number - 1) %/% 3 + 1
dframe[["combo_foods"]] <- factor(with(dframe, cheese * 1 + brassica * 2 + meat * 4 + spice * 8))

dframe[["trend"]] = 1:length(dframe$time_sleeping)
dframe[["ar_1"]] = c(dframe$time_sleeping[1], head(dframe$time_sleeping, -1))
dframe[["ar_2"]] = c(rep(dframe$time_sleeping[1], 2), head(dframe$time_sleeping, -2))
dframe[["ar_3"]] = c(rep(dframe$time_sleeping[1], 3), head(dframe$time_sleeping, -3))
dframe[["ar_4"]] = c(rep(dframe$time_sleeping[1], 4), head(dframe$time_sleeping, -4))

`1`	`model_frame <- dplyr::select(dframe, time_sleeping, ar_1, month, cheese, brassica, meat, spice)`

lm_model = lm(
  time_sleeping ~ .,
  data = model_frame
)

summary(lm_model)
noquote(sprintf("AIC: %.3f", AIC(lm_model)))

glm_model = glm(
  time_sleeping ~ .,
  data = model_frame,
  family = Gamma
)

summary(glm_model)
noquote(sprintf("AIC: %.3f", AIC(glm_model)))

wb_df <- wb_model$df.residual
wb_loglik <- wb_model$loglik[2]
lm_df <- lm_model$df.residual
lm_loglik <- logLik(lm_model)[1]
glm_df <- glm_model$df.residual
glm_loglik <- logLik(glm_model)[1]

wb_lm_lrt_stat <- 2 * (wb_loglik - lm_loglik)
wb_glm_lrt_stat <- 2 * (wb_loglik - glm_loglik)
lm_glm_lrt_stat <- 2 * (lm_loglik - glm_loglik)

wb_lm_df_diff <- abs(wb_df - lm_df)
wb_glm_df_diff <- abs(wb_df - glm_df)
lm_glm_df_diff <- abs(lm_df - glm_df)

wb_lm_p_value <- pchisq(wb_lm_lrt_stat, df = wb_lm_df_diff, lower.tail = FALSE)
wb_glm_p_value <- pchisq(wb_glm_lrt_stat, df = wb_glm_df_diff, lower.tail = FALSE)
lm_glm_p_value <- pchisq(lm_glm_lrt_stat, df = lm_glm_df_diff, lower.tail = FALSE)

noquote(sprintf("WB > LM, Likelihood Ratio Test Statistic: %.3f, p-value: %.3f", wb_lm_lrt_stat, wb_lm_p_value))
noquote(sprintf("WB > GLM, Likelihood Ratio Test Statistic: %.3f, p-value: %.3f", wb_glm_lrt_stat, wb_glm_p_value))
noquote(sprintf("LM > GLM, Likelihood Ratio Test Statistic: %.3f, p-value: %.3f", lm_glm_lrt_stat, lm_glm_p_value))

wb_fitted <- predict(wb_model, type = "response")
wb_residuals <- residuals(wb_model, type = "deviance")

plot(wb_fitted, wb_residuals,
     xlab = "Fitted Values",
     ylab = "Deviance Residuals",
     main = "Residuals vs Fitted Values (Weibull Model)",
     pch = 20)
spline_fit <- smooth.spline(wb_fitted, wb_residuals)
# Add the fitted spline line to the plot
lines(spline_fit, col = "red", lwd = 1, lty=5)

sqrt_abs_residuals <- sqrt(abs(wb_residuals))
plot(wb_fitted, sqrt_abs_residuals,
     xlab = "Fitted Values",
     ylab = "Sqrt(|Residuals|)",
     main = "Scale-Location Plot (Weibull Model)",
     pch = 20)
spline_fit <- smooth.spline(wb_fitted, sqrt_abs_residuals)
# Add the fitted spline line to the plot
lines(spline_fit, col = "red", lwd = 1, lty=5)

1
2

qqnorm(wb_residuals, main = "Q-Q Plot of Deviance Residuals (Weibull Model)")
qqline(wb_residuals, col = "red")

ar_1_pm <- c(-1.0, -0.5, 0.0, 0.5, 1.0)
sleep_night_before <- exp(wb_model[["coefficients"]][["(Intercept)"]]) + ar_1_pm


newdata_base_case = data.frame(
  month = "Apr",
  ar_1 = sleep_night_before,
  cheese = FALSE,
  brassica = FALSE,
  meat = FALSE,
  spice = FALSE
  )

base_predicted_sleep <- predict(wb_model, newdata = newdata_base_case, type = "response")

for (i in seq_along(sleep_night_before)) {
  cat(noquote(sprintf("Night before %s -> predicted time %s\n", convert_hms(sleep_night_before[i]), convert_hms(base_predicted_sleep[i]))))
}

result_frame = cbind(
  time_sleeping = dframe$time_sleeping,
  wb_model_preds = predict(wb_model, type = "response"),
  lm_model_preds = predict(lm_model, type = "response"),
  glm_model_preds = predict(glm_model, type = "response")
  )

y_min = min(result_frame)
y_max = max(result_frame)

result_frame <- cbind(result_frame, trend = dframe$trend)

plot_function <- function(series, series_name, fill, color) {
  ggplot(data = result_frame, aes(x = trend, y = {{series}})) +
  geom_point(shape = 21, fill = fill, color = color, size = 3, alpha = 0.8) +
  labs(title = series_name, x = "Month", y = "Time Sleeping (hours)") +
  scale_x_continuous(breaks = month_positions, labels = month_name) +
  scale_y_continuous(limits = c(y_min, y_max)) + 
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 90, hjust = 1))
}

p1 <- plot_function(time_sleeping, "Sample Data", "steelblue", "black")
p2 <- plot_function(wb_model_preds, "Weibull Model Predictions", "#6A0DAD", "black")
p3 <- plot_function(lm_model_preds, "Linear Model Predictions", "#CC503E", "black")
p4 <- plot_function(glm_model_preds, "Gamma Model Predictions", "#2E8B57", "black")

# Combine plots in a 2x2 grid using patchwork
(p1 | p2) / (p3 | p4)

shape <- 1 / wb_model$scale

granularity <- 1000

paste_plus <- function(num) {
  ifelse(num > 0, paste0("+", num), num)
}

base_case_names <- paste0("t", paste_plus(ar_1_pm))

get_probs <- function(linear_predictor, case_names) {
  scale <- exp(linear_predictor)  # Scale parameter for each observation
  time <- seq(min(dframe$time_sleeping), max(dframe$time_sleeping), length.out = granularity) # smooth curve over complete time range
  surv_probs_tibble <- tibble(time = time)
  
  for (i in seq_along(scale)) {
    surv_probs_tibble <- surv_probs_tibble %>%
      mutate(!!case_names[i] := exp(-(time / scale[i])^shape))
  }
  
  return(surv_probs_tibble)
}


surv_plot_func <- function(surv_probs) {
  
  # Reshape the data to long format for ggplot
  surv_probs_long <- surv_probs %>%
    pivot_longer(cols = -time, names_to = "observation", values_to = "survival_prob")
  
  surv_probs_long$observation <- factor(surv_probs_long$observation, levels = unique(surv_probs_long$observation))
  
  # Create the ggplot object
  p <- ggplot(surv_probs_long, aes(x = time, y = survival_prob, color = observation)) +
    geom_line() +
    geom_hline(yintercept = 0.5, linetype = "dashed", color = "grey", alpha = 0.75) + 
    labs(x = "Time", y = "Survival Probability", title = "Survival Curves for Controlled Observations (Weibull Model)") +
    theme_minimal()  # Minimal theme for a clean look
  
  return(p)
}

get_median_survival <- function(scale) {
  # expecting linear predictor on log scale
  exp(scale) * (log(2))^(1 / shape)
}

linear_predictor <- predict(wb_model, newdata = newdata_base_case, type = "lp")
surv_plot_func(get_probs(linear_predictor, base_case_names))

median_surv <- get_median_survival(linear_predictor)
names(median_surv) <- base_case_names
for (i in seq_along(sleep_night_before)) {
  cat(noquote(sprintf("Night before %s -> predicted median %s\n", convert_hms(sleep_night_before[i]), convert_hms(median_surv[i]))))
}

newdata_food_trigger_cases = data.frame(
  month = "Apr",
  ar_1 = exp(wb_model[["coefficients"]][["(Intercept)"]]),
  cheese = c(FALSE, TRUE, FALSE, FALSE, FALSE),  
  brassica = c(FALSE, FALSE, TRUE, FALSE, FALSE),
  meat = c(FALSE, FALSE, FALSE, TRUE, FALSE),
  spice = c(FALSE, FALSE, FALSE, FALSE, TRUE)
  )

food_trigger_case_names <- c("none", "cheese", "brassica", "meat", "spice")

linear_predictor <- predict(wb_model, newdata = newdata_food_trigger_cases, type = "lp")
surv_plot_func(get_probs(linear_predictor, food_trigger_case_names))

median_surv <- get_median_survival(linear_predictor)
names(median_surv) <- food_trigger_case_names
for (i in seq_along(sleep_night_before)) {
  cat(noquote(sprintf("Food trigger %s -> predicted median %s\n", food_trigger_case_names[i], convert_hms(median_surv[i]))))
}

newdata_food_trigger_cases = data.frame(
  month = month_name,
  ar_1 = exp(wb_model[["coefficients"]][["(Intercept)"]]),
  cheese = FALSE,  
  brassica = FALSE,
  meat = FALSE,
  spice = FALSE
  )

linear_predictor <- predict(wb_model, newdata = newdata_food_trigger_cases, type = "lp")
surv_plot_func(get_probs(linear_predictor, month_name))

median_surv <- get_median_survival(linear_predictor)
names(median_surv) <- month_name
for (i in seq_along(month_name)) {
  cat(noquote(sprintf("Month %s -> predicted median %s\n", month_name[i], convert_hms(median_surv[i]))))
}

Round Peg in a Square Hole

Fri, 08 Jan 2021 00:00:00 +0000

Introduction #

This post is a prologue to a forthcoming post on survival analytics and featucres the exploratory analysis of a self-generated data set that I will use for another demonstration post in the next few weeks.

Let me give you the background quickly, because I will go deeper with the next post: I have a really chronic problem with insomnia, since I was a teenager. I had a sense that certain foods were triggering bad nights’ sleep so I kept a food diary of what I’d eaten for dinner and recorded my quality sleep hours with a fitness tracker for a year.

This post will show a couple of visual charts, and won’t dwell on the findings so much. I have a different point I want to make today, which was triggered by a look through some Tableau examples.

Pies, Radial and Circular Charts - Are They Always a Poor Choice? #

It’s an old debate, which I thought was settled a long time ago. Edward Tufte makes the point so well in his book “The Visual Display of Quantitative Information.” Humans struggle to compare angles and arc lengths with any kind of estimation accuracy. They often take up excessive screen real estate because most rendering tools require a square space of diameter^2, which is then often hard to fit in around other elements.

Let’s take a look at the examples that triggered me to write this post and then I’ll show my personal dataset with a similar treatment:

Common Birthdays Dataset #

The circular version:

That’s a very cute Tableau Viz with a lot of care and attention on aesthetics. The circular layout is very eye-catching and provides a visual harmony with the large text information block on the right, which left a free square area for the viz itself. The placement of filter drop down in the centre is an excellent use of the available white space.

Based on aesthetic choices alone, I can see why the circular layout was used, but my data analyst eye wants every pixel choice to mean something. Size matters, so why are the areas for days in January so much bigger than December? The answer is no reason at all apart from the geometry of a circle.

Secondly, there’s the matter of eye-tracking and pattern discovery. The science of data communication tells me that I am probably in the majority when I am struggling to follow the cells around the circle. I’m trying to identify the months with the largest population density. Is it September or October? My eye keeps falling off the circular track.

The block version:

So, this is much more boring layout and less attractive. The author hasn’t tried to add any aesthetic elements however, so it’s not a completely fair comparison.

Nevertheless, it answers the points I made about the previous plot. No confusion over size/area of cells and horizontal pattern finding is intuitive and instantaneous.

I know which one I prefer. What do you think? Given it is so much more work to create the radial/polar transformations, is it ever worth it?

Insomnia and Food Triggers Data Set #

So, onto my new self-generated data set. I wanted to do a simple EDA to begin with so I popped the csv file into a new Tableau Viz. To test my assertions about radial charts, I built a dashboard with two vizualizations of the same data.

Unsurprisingly, I ended up in the same place with my thoughts as with the Birthday dataset. Speaking purely aesthetically, the radial view is really eye-catching, while the bars are heinously ugly.

It’s apparent, on the other hand, that the bars chart is just easier to interpret. I couldn’t keep the scales (Time Sleeping) on the radial chart because they’re only valid for strict horizontal and veritcal bars. Dropping reference lines onto the bar chart is a breeze, while it’s a whole additional set of calculated fields for angles and lengths and careful configuration of transparency and placement for additional elements. Frankly, who has the time and the patience, especially in a proprietary tool like Tableau where options are limited (I’d prefer to program it out in R graphics).

Conclusions #

Circular charts look nice and show that you’re clever enough to set up the transformation from cartesian to polar co-ordinates, but that serves one purpose; to highlight the analyst’s prowess, and not to help the consumer of the data product.

To put it another way, just because you can, doesn’t mean that you should.

Statistical Learning for Heart Disease Diagnosis

Thu, 05 Sep 2019 00:00:00 +0000

Author’s Note (October 2020): In preparation for work on my published research paper Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences, I wanted to develop some familiarity with medical statistics and diagnostic tools. I decided to take a fairly well-known medical data set and try something different. Instead of performing a straightforward classification model training as would be typical, I took a deep exploratory dive. My intention was to see if I could tease out some intuition as to what were the predictive patterns in the data. The exercise evolved into a potential approach to developing a statistical diagnostic tool, for similarly distributed data. My original report is reproduced below.

Advanced Exploratory Analysis of the UCI Heart Data #

Introduction #

We will conduct an analysis of the UCI Machine Learning Repository heart data set. This data set is often used for demonstrating classification methods. The target variable is usually AtheroscleroticHDisease, which indicates the presence or absence of pathologies of the blood vessels that supply the heart muscle itself. We will do something slightly different here and demonstrate several unsupervised machine learning methods to perform a thorough exploratory analysis. Exploratory analysis is essential for any serious data analytic work in order to develop an intuition about the data, identify the most important independent variables and determine the most appropriate confirmatory and hypothesis tests.

Note, we defer printing the source code until the end of the document, except where we have modified the data. In that case, the code is shown so the reader can understand the actions in context

Research Questions #

Using descriptive statistics, can we identify patterns among the independent variables?
Can we use unsupervised learning techniques to deepen our understanding of patterns and relationships in the data?

Analytic Strategy #

We will proceed in two phases:

An initial exploratory analysis with descriptive statistics:
1. to assess the data quality and perform any necessary cleansing.
2. to develop an intiution of the distributions and interactions between variables
3. this phase will comprise of visual analytics with density plots, box plots, fourfold plots and mosaic plots.
A comparison of various unsupervised learning methods:
1. matrix decomposition methods: PCA and Correspondence Analysis.
2. clustering methods, including hierarchical and distance based methods.
3. demonstrate any link between clustering and dimension reduction
4. finally, compare cluster membership with the known target variable; is there alignment and could cluster membership be an indicator of the presence of heart disease in an individual in the data set?

Note, this is a little different then a typical classification model training. Rather than performing a train/test split, we simply exclude the target variable until after conducting the unsupervised learning techniques.

Initial Exploratory Analysis #

The data dictionary provides the following information:

name	type	notes
AtheroscleroticHDisease	factor	Presence of heart disease
Age	integer	Age in years
Sex	factor	The two accepted levels at the time of data collection
ChestPain	factor	Presence/type of chest pain
RestBP	integer	Resting blood pressure mm Hg
Chol	integer	Serum cholesterol mg/dl
Fbs	factor	Fasting blood sugar
RestECG	factor	Resting electrocardiograph results
MaxHR	integer	Maximum heart rate during exercise
ExAng	factor	Exercise induced angina
Oldpeak	factor	ST depression exercise relative to rest
Slope	factor	Slope of peak ST exercise segment
Ca	small integer	Number of major vessels under fluoroscopy
Thal	factor	No description given

The data is imported from a csv file. There are 303 rows and 15 columns. Below is the head and summary.

## # A tibble: 303 × 15
##        X   Age   Sex ChestPain    RestBP  Chol   Fbs RestECG MaxHR ExAng Oldpeak
##    <int> <int> <int> <chr>         <int> <int> <int>   <int> <int> <int>   <dbl>
##  1     1    63     1 typical         145   233     1       2   150     0     2.3
##  2     2    67     1 asymptomatic    160   286     0       2   108     1     1.5
##  3     3    67     1 asymptomatic    120   229     0       2   129     1     2.6
##  4     4    37     1 nonanginal      130   250     0       0   187     0     3.5
##  5     5    41     0 nontypical      130   204     0       2   172     0     1.4
##  6     6    56     1 nontypical      120   236     0       0   178     0     0.8
##  7     7    62     0 asymptomatic    140   268     0       2   160     0     3.6
##  8     8    57     0 asymptomatic    120   354     0       0   163     1     0.6
##  9     9    63     1 asymptomatic    130   254     0       2   147     0     1.4
## 10    10    53     1 asymptomatic    140   203     1       2   155     1     3.1
## # ℹ 293 more rows
## # ℹ 4 more variables: Slope <int>, Ca <int>, Thal <chr>,
## #   AtheroscleroticHDisease <chr>

##        X              Age             Sex          ChestPain        
##  Min.   :  1.0   Min.   :29.00   Min.   :0.0000   Length:303        
##  1st Qu.: 76.5   1st Qu.:48.00   1st Qu.:0.0000   Class :character  
##  Median :152.0   Median :56.00   Median :1.0000   Mode  :character  
##  Mean   :152.0   Mean   :54.44   Mean   :0.6799                     
##  3rd Qu.:227.5   3rd Qu.:61.00   3rd Qu.:1.0000                     
##  Max.   :303.0   Max.   :77.00   Max.   :1.0000                     
##                                                                     
##      RestBP           Chol            Fbs            RestECG      
##  Min.   : 94.0   Min.   :126.0   Min.   :0.0000   Min.   :0.0000  
##  1st Qu.:120.0   1st Qu.:211.0   1st Qu.:0.0000   1st Qu.:0.0000  
##  Median :130.0   Median :241.0   Median :0.0000   Median :1.0000  
##  Mean   :131.7   Mean   :246.7   Mean   :0.1485   Mean   :0.9901  
##  3rd Qu.:140.0   3rd Qu.:275.0   3rd Qu.:0.0000   3rd Qu.:2.0000  
##  Max.   :200.0   Max.   :564.0   Max.   :1.0000   Max.   :2.0000  
##                                                                   
##      MaxHR           ExAng           Oldpeak         Slope      
##  Min.   : 71.0   Min.   :0.0000   Min.   :0.00   Min.   :1.000  
##  1st Qu.:133.5   1st Qu.:0.0000   1st Qu.:0.00   1st Qu.:1.000  
##  Median :153.0   Median :0.0000   Median :0.80   Median :2.000  
##  Mean   :149.6   Mean   :0.3267   Mean   :1.04   Mean   :1.601  
##  3rd Qu.:166.0   3rd Qu.:1.0000   3rd Qu.:1.60   3rd Qu.:2.000  
##  Max.   :202.0   Max.   :1.0000   Max.   :6.20   Max.   :3.000  
##                                                                 
##        Ca             Thal           AtheroscleroticHDisease
##  Min.   :0.0000   Length:303         Length:303             
##  1st Qu.:0.0000   Class :character   Class :character       
##  Median :0.0000   Mode  :character   Mode  :character       
##  Mean   :0.6722                                             
##  3rd Qu.:1.0000                                             
##  Max.   :3.0000                                             
##  NA's   :4

Data Cleansing - First Pass #

There are several obvious problems with the raw data set. X is an index number column and should be removed. AtheroscleroticHDisease is too much to type, so we will change the name. Several variables are coded as numeric but are factors; Sex, for example. The documentation here is informative and guides the following corrections.

# recoding
heart <- heart %>% select(-X) %>%
  rename(HDisease = AtheroscleroticHDisease) %>%
  mutate(Sex = factor(ifelse(Sex == 1, "M", "F"))
         , Fbs = factor(ifelse(Fbs == 1, ">120", "<=120"))
         , RestECG = factor(ifelse(RestECG == 0, "normal"
                                   , "abnormal")) # there are only 4 valued at 1, we'll reduce to just two levels
         , ExAng = factor(ifelse(ExAng == 1, "Yes", "No"))
         , Slope = factor(ifelse(Slope == 1, "down", "level")) # there few valued at 3, we'll reduce to just two levels
         , Ca = factor(Ca, ordered = TRUE))

# identify numerical and categorical variables for later use
classes <- sapply(heart, function(x) class(x)[1])
num_vars <- names(classes)[classes %in% c("integer", "numeric")]
cat_vars <- names(classes)[!(classes %in% c("integer", "numeric"))]

Identify Missing Values #

There is a very small number of missing values, which we would like to impute based on row-wise information. The figure demonstrates that there are no instances that share missingness in both the variables involved. The non-parametric nearest neighbours imputation is a reasonable choice, as it makes no assumptions about the data. This will be executed shortly.

Univariate Distributions #

Next, we show a kernel density plot of each numeric variables.

## Skew

1
2

##        Age     RestBP       Chol      MaxHR    Oldpeak 
## -0.2069951  0.6990596  1.1242853 -0.5321391  1.2571761

A visual inpection of the above plots reveals skew and non-normality in the Oldpeak and Chol variables. This is confirmed by a statistical test for skew, where values of skew >1 are severe. The skew in the Chol variable appears to be caused by an outlier. Briefly researching this matter online, it is apparent that a reading of >200 for cholesterol is already considered extremely high and the problematic reading in our dataset is nearly 600. On the other hand, the Oldpeak variable is systemically skewed. Power transformations are unsuitable for this variable because of the prevalence of zero values, so this will be left as is.

Data Cleansing - Second Pass #

We will perform the following actions:

Set the outlying Chol value to missing (NA),
Impute missing values for Chol, Ca and Thal, and
Scale all the variables between [0,1] - this is to support the distance based clustering techniques that we will use later.

# modified min/max functions
minval <- function(x) min(x, na.rm = TRUE)
maxval <- function(x) max(x, na.rm = TRUE)

# change outlier to missing
heart$Chol[which.max(heart$Chol)] <- NA

# scale to [0, 1]
for (nv in num_vars) {
  this_nv <- pull(heart, nv) 
  if(minval(this_nv) < 0) {
    heart[[nv]] <- (this_nv + abs(minval(this_nv))) / 
      (maxval(this_nv) + abs(minval(this_nv)))  
  } else {
    heart[[nv]] <- (this_nv - minval(this_nv)) / 
      (maxval(this_nv) - minval(this_nv))
  }
}

# impute missing - VIM package. Median is the default function. Pick a moderately large k
heart <- as_tibble(kNN(heart, c("Chol", "Ca", "Thal"), imp_var = FALSE, k=11))

# for later use
heart_num <- select(heart, num_vars)
heart_cat <- select(heart, cat_vars)

`1`	`## Recoded data set`

## # A tibble: 303 × 14
##      Age Sex   ChestPain    RestBP  Chol Fbs   RestECG MaxHR ExAng Oldpeak Slope
##    <dbl> <fct> <chr>         <dbl> <dbl> <fct> <fct>   <dbl> <fct>   <dbl> <fct>
##  1 0.708 M     typical       0.481 0.368 >120  abnorm… 0.603 No     0.371  level
##  2 0.792 M     asymptomatic  0.623 0.550 <=120 abnorm… 0.282 Yes    0.242  level
##  3 0.792 M     asymptomatic  0.245 0.354 <=120 abnorm… 0.443 Yes    0.419  level
##  4 0.167 M     nonanginal    0.340 0.426 <=120 normal  0.885 No     0.565  level
##  5 0.25  F     nontypical    0.340 0.268 <=120 abnorm… 0.771 No     0.226  down 
##  6 0.562 M     nontypical    0.245 0.378 <=120 normal  0.817 No     0.129  down 
##  7 0.688 F     asymptomatic  0.434 0.488 <=120 abnorm… 0.679 No     0.581  level
##  8 0.583 F     asymptomatic  0.245 0.784 <=120 normal  0.702 Yes    0.0968 down 
##  9 0.708 M     asymptomatic  0.340 0.440 <=120 abnorm… 0.580 No     0.226  level
## 10 0.5   M     asymptomatic  0.434 0.265 >120  abnorm… 0.641 Yes    0.5    level
## # ℹ 293 more rows
## # ℹ 3 more variables: Ca <ord>, Thal <chr>, HDisease <chr>

##       Age         Sex      ChestPain             RestBP            Chol       
##  Min.   :0.0000   F: 97   Length:303         Min.   :0.0000   Min.   :0.0000  
##  1st Qu.:0.3958   M:206   Class :character   1st Qu.:0.2453   1st Qu.:0.2921  
##  Median :0.5625           Mode  :character   Median :0.3396   Median :0.3952  
##  Mean   :0.5300                              Mean   :0.3556   Mean   :0.4112  
##  3rd Qu.:0.6667                              3rd Qu.:0.4340   3rd Qu.:0.5103  
##  Max.   :1.0000                              Max.   :1.0000   Max.   :1.0000  
##     Fbs          RestECG        MaxHR        ExAng        Oldpeak      
##  <=120:258   abnormal:152   Min.   :0.0000   No :204   Min.   :0.0000  
##  >120 : 45   normal  :151   1st Qu.:0.4771   Yes: 99   1st Qu.:0.0000  
##                             Median :0.6260             Median :0.1290  
##                             Mean   :0.6001             Mean   :0.1677  
##                             3rd Qu.:0.7252             3rd Qu.:0.2581  
##                             Max.   :1.0000             Max.   :1.0000  
##    Slope     Ca          Thal             HDisease        
##  down :142   0:179   Length:303         Length:303        
##  level:161   1: 66   Class :character   Class :character  
##              2: 38   Mode  :character   Mode  :character  
##              3: 20                                        
##                                                           
##

Bivariate Distributions #

We would now like to look for colinearity or correlation among the predictors. A simple bivariate correlation analysis is conducted.

Careful inspection of the splom shows a clear elevation of Oldpeak in the HDisease = Yes group. There is a potentially interesting interaction between Age and MaxHR, which is the pair with the strongest correlation: Specifically, MaxHR is more correlated with Age in the HDisease = No group, whereas MaxHR is slightly lower whatever the Age in the HDisease = Yes group. The HDisease = Yes group are formed from a narrower Age band, indicating that a particular age range could carry higher risk. The remaining continuous variables may be less informative.

Identifying Candidate Predictors #

We now generate a boxplot for each of the five continuous variables, conditioned on each of the factor variables. At this point, we include the HDisease variable to evaluate candidate predictors. This knowledge would be useful prior to developing a classification model of some kind.

The figures that follow need to be assessed row by row.

A visual instpection of the boxplots indicates that there may be some interaction between Chestpain and the three informative variables identified previously (Age, MaxHR and Oldpeak). This is also true of ExAng (exercise induced angina), Slope, Ca and Thal. We will not perform any statistical tests at this stage. Fishing for p-values with so many tests requires careful attention to the false discovery rate, and assumptions checking and diagnostics for an unwieldy number of variable combinations. It is sufficient to develop an intuition about the data and interactions.

Multivariate Counts #

Distributions among categorical variables can be assessed according to counts and proportions. The following Fourfold and Mosaic visualisations implicitly perform significance tests by means of shading residuals. Fourfold plots are suited to visualising pairs of binary variables, while mosaics can handle factors with more levels.

We can see that the presence of atherosclerotic heart disease is significantly associated with males, while a third order interaction exists such that those who are generally free from chest pain but suffer with exercise induced angina have a significant risk of atherosclerotic heart disease.

The mosaic plot shows a four way analysis, which requires some explanation to those unfamiliar with this technique. Mosaics plots may look daunting at first, but quickly become intuitive with a little practice. They visualise a recursive partition. In this case, starting on the right with HDisease, the plot canvas is split horizontally into two tiles for “Yes” and “No”. Going clockwise, the tiles are split vertically for Slope “down” and “level”. Next, the tiles are split horizontally again for all the levels of Ca, into the existing two levels of HDisease. Finally, the collection of tiles are split veritcally once more for each level of Thal into the Slope levels. At each split, the tile areas are set to be proportional to the count at each intersection of variables. Independent variables would show up as evenly spaced and evenly sized tiles, where each tile would be proportional is size/count to the marginal totals. Discontinuities in the slice lines show up when an intersection between variables contains significantly more or fewer counts than expected under the independence assumption. When this occurs, the tiles are shaded in a two level gradient: blue/red for positive/negative association compared to the independence assumption, which aids the observer in understanding positive and negative associations. The shading values are based on the residuals when compare to a non-significant $\chi^2$ test.

We can see that downward slope is most associated with the HDisease = No group and level slope for HDisease = Yes. Similarly, one or more arteries visible under fluoroscopy (Ca) are most strongly associated with HDisease = Yes. A third order interaction indicates that a combination of level Slope and reversible Thal is very strongly associated with HDisease = Yes, with fixed Thal also being strongly indicative.

Unuspervised Learning Methods #

We continue with a more advanced exploration of this data set.

PCA #

The first method we will exlore is principle components analysis (PCA) on the scaled continuous variables. PCA identifies orthogonal projections of multivariate data that capture most of the variation in the first components.

## Standard deviations (1, .., p=5):
## [1] 0.2423102 0.1803325 0.1635007 0.1514306 0.1241028
## 
## Rotation (n x k) = (5 x 5):
##                PC1        PC2        PC3        PC4        PC5
## Age     -0.6025250  0.3042504 -0.4586509  0.2202109  0.5343611
## RestBP  -0.3156058  0.4109580  0.4662090  0.5759948 -0.4270663
## Chol    -0.1892079  0.6063496  0.1497098 -0.7478426 -0.1218969
## MaxHR    0.4950597  0.3664037  0.4208113  0.1503096  0.6488367
## Oldpeak -0.5064314 -0.4864501  0.6105680 -0.1946165  0.3102009

## Importance of components:
##                           PC1    PC2    PC3    PC4     PC5
## Standard deviation     0.2423 0.1803 0.1635 0.1514 0.12410
## Proportion of Variance 0.3756 0.2081 0.1710 0.1467 0.09854
## Cumulative Proportion  0.3756 0.5837 0.7548 0.9015 1.00000

We can see from the output summary that Age and Oldpeak are loaded onto PC1 in the opposite direction to MaxHR, indicating a negative correlation. The other PCs can generally be interpreted according to further relationships among these variables, some of which we have already seen in the bivariate correlation analysis. It seems from the cumulative variance measure and the scree plot that the projection of these five numeric features onto less interpretable principal components does not offer any obvious gains; it still takes four components to capture most of the variance, nearly the same number as the raw variables. Nevertheless, we can make a biplot in an attempt to better understand the multivariate relationships.

The biplot doesn’t reveal any useful new information, so we proceed to the next technique.

Multiple Correspondence Analysis #

Multiple correspondence analysis is a dimension reduction and clustering analysis for categorical counts. Variables with strong count associations tend toward the same direction in a biplot, and we would expect some of the relationships revealed in the mosaic plot above to be evident here as well.

As with PCA, the relationships among several or many categorical variables can be mapped in two dimensions, giving fairly intiuitive results. A less well-known benefit is that categorical variables are implicitly re-coded into non-abritrary numeric values. This makes them available to use in any methods that only accept real-valued inputs, such as distance-based methods like k-means or hierarchical clustering.

## 
## Principal inertias (eigenvalues):
## 
##  dim    value      %   cum%   scree plot               
##  1      0.060363  83.7  83.7  ************************ 
##  2      0.000879   1.2  84.9                           
##  3      0.000424   0.6  85.5                           
##  4      0.000320   0.4  85.9                           
##  5      2.1e-050   0.0  85.9                           
##  6      1.2e-050   0.0  85.9                           
...

The results of this analysis are indeed rather interesting and confirm the findings of the initial explaratory analysis. The plot can be interpreted by identifying attributes that have moved in the same direction from the origin (0, 0), with particular interest in those that have clustered close together. We can see the specific values of ChestPain, Slope, Ca, ExAng and Sex that are associated with the presence of absence of heart diseease. We also have confirmation that Fbs is much less correlated with HDisease and RestECG only moderately correlated. If we were going on to do predictive modeling, this analysis would provide a stong case for excluding them. In fact, we will re-run this analysis excluding those variables to demonstrate the impact.

## 
## Principal inertias (eigenvalues):
## 
##  dim    value      %   cum%   scree plot               
##  1      0.102077  88.4  88.4  *************************
##  2      0.000790   0.7  89.1                           
##  3      0.000493   0.4  89.5                           
##  4      0.000214   0.2  89.7                           
##  5      2e-05000   0.0  89.7                           
##         -------- -----                                 
##  Total: 0.115476                                       
...

We are not displaying the arrows this time because the labels themselves are rather crowded and hard enough to read without the extra visual clutter. We can see that the clusters are very pronounced and we can once again identify a handful of less informative attributes. For example, Thal=fixed and ChestPain=typical represent small minorities in the way these variables are distributed.

We can augment this analysis by discretizing the numeric variables and including them as well. This will be done by binning into low and high levels.

hilo <- function(x, ind) {
  cut(x, breaks=c(quantile(pull(heart, ind), probs = c(0, 0.5, 1))),
      labels = c("lo", "hi")
      )
}
binned <- as_tibble(sapply(num_vars
                , function(ind) sapply(select(heart, ind)
                                       , hilo
                                       , ind = ind)))
names(binned) <- num_vars
heart_bins <- bind_cols(heart_cat, binned)

## 
## Principal inertias (eigenvalues):
## 
##  dim    value      %   cum%   scree plot               
##  1      0.056110  74.6  74.6  ***********************  
##  2      0.001968   2.6  77.2  *                        
##  3      0.001005   1.3  78.6                           
##  4      0.000532   0.7  79.3                           
##  5      0.000158   0.2  79.5                           
##  6      4.9e-050   0.1  79.5                           
##  7      2.3e-050   0.0  79.6                           
##         -------- -----                                 
##  Total: 0.075208                                       
## 
...

This plot takes some time to interpret, but a careful look further confirms results of our previous analyses. The lowest MaxHR values are associated with heart disease and the highest values with absence of disease. Increasing Age is generally associated with presence of heart disease. Increasing Chol is associated, but only very moderately (the Chol attribute levels are not separated widely on the horizontal dimension). The same is true for RestBP. Let us re-run this once more, excluding these two less correlated variables. We also exclude the HDisease label as we do not want it to influence the resulting numerical values.

## 
## Principal inertias (eigenvalues):
## 
##  dim    value      %   cum%   scree plot               
##  1      0.062594  70.5  70.5  ************************ 
##  2      0.001733   2.0  72.5  *                        
##  3      0.000913   1.0  73.5                           
##  4      0.000390   0.4  74.0                           
##  5      0.000266   0.3  74.3                           
##  6      6.5e-050   0.1  74.3                           
##  7      2.8e-050   0.0  74.4                           
##         -------- -----                                 
##  Total: 0.088746                                       
...

## $rows
##      Dim1 Dim2
## <NA>   NA   NA
## 
## $cols
##                               Dim1         Dim2
## Sex:F                   0.19555307  0.105634316
## Sex:M                  -0.07492566 -0.050559816
## ChestPain:asymptomatic -0.27881108 -0.010815584
## ChestPain:nonanginal    0.25208525  0.032753180
## ChestPain:nontypical    0.45068134 -0.051398877
## ChestPain:typical       0.01173958  0.048169595
## ExAng:No                0.20642820  0.019043790
...

This multiple correspondence analysis provides two very well differentiated clusters of attributes. This result was generated in the absence of the HDisease variable, yet we have plenty of reason to believe that these engineered variables we could are strongly predictive. Nearly all the inertia (variance) is captured on the first dimension, we can use the one-dimensional coordinate values from this dimension as numerical proxies for the categorical attributes in the analyses to follow.

# res is the plot object created by (res <- plot(...))
# the last 6 rows are the three continuous vars, and we just want to convert categorical to real valued.
coords <- data.frame(res$cols, heart_mca$factors)[-(20:25), ]

# We don't need the HDisease feature as we will see how well the clusters match these labels.
# We'll add a column of missing values for each of the other categorical variables
heart_real <- select(heart_num, -Chol, -RestBP)
for(cv in names(select(heart_cat, -HDisease))) { # Fbs and RestECG were removed already
  heart_real <- mutate(heart_real, !!cv := NA)
}
# Now we'll insert the value from Dim 1 for each level of each factor
for(i in 1:nrow(coords)) {
  fac <- as.character(coords$factor[i])
  lev <- as.character(coords$level[i])
  heart_real <- mutate(heart_real, !!fac := ifelse(pull(heart, fac) == lev, coords$Dim1[i], pull(heart_real, fac)))
}

Distance-based Clustering #

Now that we have selected our variables of interest and converted categorical values to non-arbitrary real-values, we can continue with distance based clustering. The following visualisation is a map of all the Euclidean distances between the points.

This visualisation provides a useful confirmation of the clustering tendency in this data. We can also compute the Hopkins statistic H, which tests the null hypothesis that the data has come from a uniform distribution and is distributed as $H \sim \mathit{Beta}(n, n)$ where n is the number of samples used to calculate the statistic. Values of H 0-0.3 indicate regularly-spaced data. Values around 0.5 indicate random data. Values 0.7-1 indicate clustered data.

1
2

H <- hopkins::hopkins(heart_real)
H

`1`	`## [1] 0.9987419`

This H value indicates very well clustered data.

K-medoids Clustering #

K-medoids searches for k archetypal or representative instances, called medoids that act as the cluster centres. Each non-medoid instance is assigned to its nearest medoid. The algorithm proceeds by swapping medoid and non-medoid points, accepting a swap that decreases the sum of a dissimilarity function. K-medoids is less sensitive to outliers than the classic K-means method, so is often favoured. Another reason to prefer K-medoids is that the number k can be estimated using the silhouette method. From the knowledge we have already that these variables tend to be associated with presence or absence of heart disease, we could assume two clusters is the correct number, but it’s still worth checking using this visual inspection.

With confidence we can run the clustering with k=2:

##            Age     MaxHR   Oldpeak         Sex  ChestPain      ExAng      Slope
## [1,] 0.5416667 0.4656489 0.1935484 -0.07492566 -0.2788111 -0.3841233 -0.2447590
## [2,] 0.4375000 0.7022901 0.0000000 -0.07492566  0.2520853  0.2064282  0.3130363
##              Ca       Thal
## [1,] -0.1711661 -0.2850512
## [2,]  0.1944836  0.2757814

##    
##      No Yes
##   1  25 114
##   2 139  25

Recall that HDisease was not included in the real-valued data cpnversion nor the clustering process. Cross-tabulating the instances from the original dataset shows a “confusion matrix-like” result, indicating that the two clusters have captured the association with heart disease among the variables of interest. The cluster labels are arbitrary in this unsupervised learning process. We can set the appropriate label to the cluster id and get a full suite of diagnostics:

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Yes  No
##        Yes 114  25
##        No   25 139
##                                          
##                Accuracy : 0.835          
##                  95% CI : (0.7883, 0.875)
##     No Information Rate : 0.5413         
##     P-Value [Acc > NIR] : <2e-16         
##                                          
##                   Kappa : 0.6677         
##                                          
##  Mcnemar's Test P-Value : 1              
##                                          
##             Sensitivity : 0.8201         
##             Specificity : 0.8476         
##          Pos Pred Value : 0.8201         
##          Neg Pred Value : 0.8476         
##              Prevalence : 0.4587         
##          Detection Rate : 0.3762         
##    Detection Prevalence : 0.4587         
##       Balanced Accuracy : 0.8339         
##                                          
##        'Positive' Class : Yes            
##

We can see that if we assigned new, unlabelled points to their nearest cluster medoid, we might expect an accuracy of around 0.83

Hierarchical Clustering #

We can do something similar with hierarchical clustering. After some experimentation with different clustering methods, the “Complete” method was chosen as it produced two distinct clusters with similar numbers of members.

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction Yes  No
##        Yes 112  33
##        No   27 131
##                                           
##                Accuracy : 0.802           
##                  95% CI : (0.7526, 0.8454)
##     No Information Rate : 0.5413          
##     P-Value [Acc > NIR] : <2e-16          
##                                           
##                   Kappa : 0.6026          
##                                           
##  Mcnemar's Test P-Value : 0.5186          
##                                           
##             Sensitivity : 0.8058          
##             Specificity : 0.7988          
##          Pos Pred Value : 0.7724          
##          Neg Pred Value : 0.8291          
##              Prevalence : 0.4587          
##          Detection Rate : 0.3696          
##    Detection Prevalence : 0.4785          
##       Balanced Accuracy : 0.8023          
##                                           
##        'Positive' Class : Yes             
##

Again, we can speculate on the accuracy we would get on prediction using just the cluster membership. This yields a slightly lower overall accuracy, but an increased specificity, or True Positive Rate. As such, this method might be more suitable when designing a diagnostic tool.

We can plot dendrograms of hierarchical clusters and colour the leaves using any factor variable. This provides a visual inspection of how the factor is distributed among the clusters, starting with the HDisease variable that we might want to classify at some future time. Again, recall that this “target variable” was not used in the clustering algorithm yet we can see that it is well separated into the two clusters. Note, the colours are arbitrary. What is interesting is how they have separated so well into prevalent groupings between the two clusters.

Conclusion #

We have performed a thorough exploratory analysis of the UCI Heart Data and used unsupervised machine learning methods to provide a visual intuition of important and potentially predictive structure in the data. We presented a reasoned approach to removing non-informative features and further reducing the dimension of the problem by correspondence analysis. After this, we were able to engineer a single, highly informative cluster membership feature, with very good potential as a diagnostic (classification) tool for similarly distributed data.

Appendix #

Here you can find the source code.

library(knitr)
library(tibble)
library(readr)
library(dplyr)
library(lattice)
library(ggplot2)
library(corrplot)
library(vcd)
library(psych)
library(car)
library(VIM)
library(ca)
library(factoextra)
library(cluster)
library(clustertend)
library(caret)
library(ape)
library(hopkins)

opts_chunk$set(warning = FALSE
              , message = FALSE
              , echo = FALSE
              )
opts_template$set(
  fig.wide = list(fig.height = 4.5, fig.width = 8, fig.align='center')
  , fig.wideX = list(fig.height = 3, fig.width = 9, fig.align='center')
  , fig.relaxed = list(fig.height = 6, fig.width = 8, fig.align='center')
  , fig.hugetile = list(fig.height = 7, fig.width = 7, fig.align='center')
  , fig.bigtile = list(fig.height = 5, fig.width = 5, fig.align='center')
  , fig.tile = list(fig.height = 3, fig.width = 3, fig.align='center')
)

hook_output <- knit_hooks$get("output")
knit_hooks$set(output = function(x, options) {
  lines <- options$output.lines
  if (is.null(lines)) {
    return(hook_output(x, options))  # pass to default hook
  }
  x <- unlist(strsplit(x, "\n"))
  more <- "..."
  if (length(lines)==1) {        # first n lines
    if (length(x) > lines) {
      # truncate the output, but add ....
      x <- c(head(x, lines), more)
    }
  } else {
    x <- c(more, x[lines], more)
  }
  # paste these lines together
  x <- paste(c(x, ""), collapse = "\n")
  hook_output(x, options)
})

par(mar = c(4,3,3,1))

set.seed(142136)

# graphing themes
source("HeartTheme.R")

# load data
heart <- as_tibble(read.csv("Heart.csv"))
variables_df <- data.frame(name = c("AtheroscleroticHDisease"
                                    , "Age", "Sex", "ChestPain"
                                    , "RestBP", "Chol", "Fbs"
                                    , "RestECG", "MaxHR"
                                    , "ExAng", "Oldpeak"
                                    , "Slope", "Ca", "Thal")
                           , type = c("factor", "integer"
                                      , "factor", "factor"
                                      , "integer", "integer"
                                      , "factor", "factor"
                                      , "integer", "factor"
                                      , "factor", "factor"
                                      , "small integer", "factor")
                           , notes = c("Presence of heart disease"
                                       , "Age in years", "The two accepted levels at the time of data collection"
                                       , "Presence/type of chest pain"
                                       , "Resting blood pressure mm Hg"
                                       , "Serum cholesterol mg/dl"
                                       , "Fasting blood sugar"
                                       , "Resting electrocardiograph results"
                                       , "Maximum heart rate during exercise"
                                       , "Exercise induced angina"
                                       , "ST depression exercise relative to rest"
                                       , "Slope of peak ST exercise segment"
                                       , "Number of major vessels under fluoroscopy"
                                       , "No description given"
                                       ))
kable(variables_df)
heart
summary(heart)
# recoding
heart <- heart %>% select(-X) %>%
  rename(HDisease = AtheroscleroticHDisease) %>%
  mutate(Sex = factor(ifelse(Sex == 1, "M", "F"))
         , Fbs = factor(ifelse(Fbs == 1, ">120", "<=120"))
         , RestECG = factor(ifelse(RestECG == 0, "normal"
                                   , "abnormal")) # there are only 4 valued at 1, we'll reduce to just two levels
         , ExAng = factor(ifelse(ExAng == 1, "Yes", "No"))
         , Slope = factor(ifelse(Slope == 1, "down", "level")) # there few valued at 3, we'll reduce to just two levels
         , Ca = factor(Ca, ordered = TRUE))

# identify numerical and categorical variables for later use
classes <- sapply(heart, function(x) class(x)[1])
num_vars <- names(classes)[classes %in% c("integer", "numeric")]
cat_vars <- names(classes)[!(classes %in% c("integer", "numeric"))]
image(is.na(select(heart, Ca, Thal))
      , main = "Missing Values"
      , xlab = ""
      , ylab = "Ca, Thal"
      , xaxt = "n"
      , yaxt = "n"
      , bty = "n"
      , col = c(myPal[4]
                   , myPalDark[5])
                   )
for (nv in num_vars) {
  densityplot(~pull(heart, nv)
             , xlab = nv
             , par.settings = myLatticeTheme) %>%
      print()
}

cat("Skew")
sapply(num_vars, function(ind) skew(pull(heart, ind)))
# modified min/max functions
minval <- function(x) min(x, na.rm = TRUE)
maxval <- function(x) max(x, na.rm = TRUE)

# change outlier to missing
heart$Chol[which.max(heart$Chol)] <- NA

# scale to [0, 1]
for (nv in num_vars) {
  this_nv <- pull(heart, nv) 
  if(minval(this_nv) < 0) {
    heart[[nv]] <- (this_nv + abs(minval(this_nv))) / 
      (maxval(this_nv) + abs(minval(this_nv)))  
  } else {
    heart[[nv]] <- (this_nv - minval(this_nv)) / 
      (maxval(this_nv) - minval(this_nv))
  }
}

# impute missing - VIM package. Median is the default function. Pick a moderately large k
heart <- as_tibble(kNN(heart, c("Chol", "Ca", "Thal"), imp_var = FALSE, k=11))

# for later use
heart_num <- select(heart, num_vars)
heart_cat <- select(heart, cat_vars)
cat("Recoded data set")
heart
summary(heart)
corrplot(cor(heart_num), order="AOE", type="upper"
        , col = myPal.rangeDiv(10)
        , tl.pos="d", tl.cex = 1, tl.col = myPalDark[1]
        , method = "number", number.cex = 1.5)
corrplot(cor(heart_num), add=TRUE, type = "lower"
        , col = myPal.rangeDiv(10)
        , method = "ellipse", order = "AOE", diag = FALSE
        , tl.pos="n", cl.pos = "n")
trel <- myLatticeTheme
trel$plot.symbol$col <- myPal[1]
splom(~heart_num | HDisease
      , data = heart
      , diag.panel = function(x, ...){
        yrng <- current.panel.limits()$ylim
        d <- density(x, na.rm=TRUE)
        d$y <- with(d, yrng[1] + 0.95 * diff(yrng) * y / max(y) )
        panel.lines(d)
        diag.panel.splom(x, ...)
 }
      , panel = function(x, y, ...){
        panel.xyplot(x, y, ..., alpha = 0.4)
        panel.loess(x, y, ..., col = myPalDark[2], lwd = 3)
      }
      , main = "Scatterplot Matrix by HDisease Group"
      , xlab = ""
      , layout = c(2, 1)
      , pscales = 0
      , par.settings = trel)
cols <- myPal.rangeDiv(5)
names(cols) <- num_vars
  
myBoxPlots <- function(nv, cv) {
  fmla <- as.formula(paste(nv, " ~ ", cv))
  boxplot(fmla, data = heart, col = cols[nv])
}
par(mfrow=c(1,5))
for (cv in cat_vars){
  for (nv in num_vars) {
    myBoxPlots(nv, cv)
  }
}
par(mfrow=c(1,1))
fourfold(with(heart, table(HDisease, Sex))
         , color = myFourFold)
fourfold(with(heart, table(HDisease, ExAng, ChestPain))
         , color = myFourFold)
mosaic(with(heart, table(HDisease, Slope, Ca, Thal))
       , gp = shading_hsv
       , gp_args = list(h = myShading["h", ]
                        , s = c(0.75, 0)
                        , v = 1
                        , lty = 1:2
                        , line_col = c(myPalDark[5], myPalDark[4])
                        )
       )
heart_pca <- prcomp(heart_num, scale. = FALSE)
heart_pca
summary(heart_pca)
plot(heart_pca, main = "scree plot for heart pca")
biplot(heart_pca
       , xlim = c(-0.2, 0.2), ylim = c(-0.2, 0.2)
       , col = c(myPal[2], myPalDark[5]), cex=c(0.5, 1.5))
heart_mca <- mjca(heart_cat)
summary(heart_mca)
plot(heart_mca, map = "symbiplot"
     , arrows = c(FALSE, TRUE)
     , mass = c(FALSE, TRUE)
     , contrib = c("none", "relative")
     , col = c(myPalDark[5], myPalDark[4]))
heart_cat <- select(heart_cat, -Fbs, -RestECG)
heart_mca <- mjca(heart_cat)
summary(heart_mca)
plot(heart_mca, map = "symbiplot"
     , mass = c(FALSE, TRUE)
     , contrib = c("none", "relative")
     , col = c(myPalDark[5], myPalDark[4]))
hilo <- function(x, ind) {
  cut(x, breaks=c(quantile(pull(heart, ind), probs = c(0, 0.5, 1))),
      labels = c("lo", "hi")
      )
}
binned <- as_tibble(sapply(num_vars
                , function(ind) sapply(select(heart, ind)
                                       , hilo
                                       , ind = ind)))
names(binned) <- num_vars
heart_bins <- bind_cols(heart_cat, binned)
heart_mca <- mjca(heart_bins)
summary(heart_mca)
plot(heart_mca, map = "symmetric"
     , mass = c(FALSE, TRUE)
     , contrib = c("none", "relative")
     , col = c(myPalDark[5], myPalDark[4]))
heart_mca <- mjca(select(heart_bins, -Chol, -RestBP, -HDisease))
summary(heart_mca)
(res <- plot(heart_mca, map = "symmetric"
     , mass = c(FALSE, TRUE)
     , contrib = c("none", "relative")
     , col = c(myPalDark[5], myPalDark[4])))
# res is the plot object created by (res <- plot(...))
# the last 6 rows are the three continuous vars, and we just want to convert categorical to real valued.
coords <- data.frame(res$cols, heart_mca$factors)[-(20:25), ]

# We don't need the HDisease feature as we will see how well the clusters match these labels.
# We'll add a column of missing values for each of the other categorical variables
heart_real <- select(heart_num, -Chol, -RestBP)
for(cv in names(select(heart_cat, -HDisease))) { # Fbs and RestECG were removed already
  heart_real <- mutate(heart_real, !!cv := NA)
}
# Now we'll insert the value from Dim 1 for each level of each factor
for(i in 1:nrow(coords)) {
  fac <- as.character(coords$factor[i])
  lev <- as.character(coords$level[i])
  heart_real <- mutate(heart_real, !!fac := ifelse(pull(heart, fac) == lev, coords$Dim1[i], pull(heart_real, fac)))
}
distance <- get_dist(heart_real)
myPalDiv <- myPal.rangeDiv(3)
fviz_dist(distance, show_labels = FALSE
          , gradient = list(low = myPalDiv[1], mid = myPalDiv[2], high = myPalDiv[3]))
H <- hopkins::hopkins(heart_real)
H
fviz_nbclust(heart_real, pam, method = "silhouette"
             , k.max = 5, linecolor = myPalDark[4]) +
  myGgTheme()
pam2_clus <- pam(heart_real, k = 2, diss = FALSE, metric = "euclidean")
pam2_clus$medoids # translate back
table(pam2_clus$cluster, heart$HDisease)
pam_factor <- factor(ifelse(pam2_clus$cluster == 1, "Yes", "No"))
hd_factor <- relevel(factor(heart$HDisease), "Yes")
cm <- confusionMatrix(pam_factor, hd_factor)
cm
fviz_cluster(pam2_clus, heart_real
             , ellipse.type = "convex"
             , geom = "point"
             , palette = c(myPalDark[4], myPalDark[5])) +
  myGgTheme()
hclus <- hclust(dist(heart_real)
                        , method="complete")
num_clus <- 2


clus_factor <- relevel(factor(ifelse(cutree(hclus, num_clus) == 1, "Yes", "No")), "Yes")
cm <- confusionMatrix(clus_factor, hd_factor)
cm
hclusPlot <- function(x, n, df, colourBy) {

  labelColours <- myPalContrasts[factor(df[[colourBy]])]

  # phylogenic tree plot
  plot(as.phylo(x)
     , direction = "downwards"
     , tip.color = myPalContrasts[factor(df[[colourBy]])]
     , main = paste(x$method, "method\nleaf colours by" , colourBy))
  rect.hclust(x, k=n, border = "red")
}

for (nm in names(heart_real)[!names(heart_real) %in% c("Age", "MaxHR")]) {
  hclusPlot(hclus, num_clus, heart, nm)
}

Freelancing In Paradise

Thu, 04 Aug 2016 00:00:00 +0000

I took a career break while I was still living in the tropics and had the chance to do some freelance work in between SCUBA dives. One of my first gigs was to help a client to evaluate their sales target setting process. Their team was split into two groups and some rumours were circulating that one group had their targets set too easy. I was able to recommend some improvements based on some very simple statistical tests.

With the client’s permission, I’ve posted a brief write-up. This is a great demonstration of how easy it is to run Bayesian equivalents of standard statistical tests, using the BayesianFirstAid package by Rasmus Bååth.

Statistical Analysis of Sales Data #

Introduction #

The client’s sales department is organised into groups A and B. Members of Group B are set much lower quotas (sales targets) on average, reflecting an added difficulty to sell to group B customers.

Business Problem #

Is there any difference performance between the two groups? What (if any) anomalies are present? Make recommendations based on the findings.

Executive Summary #

Members of Group B are much more sensitive to quota size, with diminishing chances of meeting their quota as quota increased over a very small range. This appears to align with the additional challenge of selling to this customer segment. There is some evidence of a disparity between the two groups, with Group B members more likely to struggle to quotas as quotas increase. The same is not true of Group A and their chances of meeting quota.

Methodology #

I reviewed the descriptive statistics before analysing the groups using Bayesian statistical tests. Bayesian methods are preferred for this analysis because the data is a complete census (the entire sales department). The interpretation is more intuitive and I do not intend to make inferences about a larger population. Furthermore, checking assumptions and diagnostics for packaged Bayesian models is much less time consuming.

Descriptive #

The following listing shows the top 6 rows of the data file to show what a simple data set I was working with. The standard descriptive summaries are shown in Appendix A so I can jump directly to the more interesting parts of the investigation.

## # A tibble: 6 × 5
##   Group Attainment KSales KQuota MetTarget
##   <fct>      <dbl>  <dbl>  <dbl> <lgl>    
## 1 A          104.    33.7   32.4 TRUE     
## 2 A           92.6   25.0   26.9 FALSE    
## 3 A          109.    29.9   27.4 TRUE     
## 4 A          107.    24.2   22.6 TRUE     
## 5 A           92.9   23.2   25.0 FALSE    
## 6 A           98.2   24.7   25.1 FALSE

Initial Exploration and Impressions #

The obvious place to start the investigation is with a correlation analysis between quota set and sales achieved. I also visualise the relationship and colour according to group membership.

`1`	`## Quota to Sales Correlation`

`1`	`## [1] 0.9776268`

As expected, given the brief, Group B have lower mean quota and sales than Group A. There appears to be a very strong correlation between quota and sales. I asked the client to consider what this correlation implies. Does the person setting the quota (targets) at the start of the sales year have an uncanny grasp of the annual sales process? Are sales team members working up to the quotas but taking their foot off the gas when they know they’ve qualified for a bonus? If so, could a stretch target change the outcomes? These are causal questions that I could not possibly answer with this simple data set but it led to some interesting follow up discussions.

Differences in Performance Within Groups #

I considered the two groups separately, given the known difference between groups already described.

For those who made their targets compared to those who didn’t, it was useful to know how much their quotas were influential. A boxplot helps to visualise this for the two groups:

In Group A, there is no real difference in the mean Quotas ($\approx$ 0.1K sales quota units) between those who met their target and those who didn’t. For Group B, this difference is notably $\approx$ -2.36K sales quota units i.e. somewhat lower among those who met their target than those who did not.

However, with such a large range of quotas (6.81, 32.32), is this difference important? A Bayesian t-test can help to answer this question. For comparison, I show the test results for both groups.

Bayesian t-test Group A #

The bottom two charts show the posterior (distribution) of the mean difference (Group A: Met Target vs Group A: Target not Met). I can see the posterior is centred very close to zero difference.

Bayesian t-test Group B #

The bottom two charts show the posterior distribution of the mean difference (Group B: Met Target vs Group B: Target not Met). I can see the posterior is centred at -2.27 which aligns with the above box plot. The effect size (relative to the variance) is considered medium at -0.41. Zero is inside the credible interval, but there is a $\approx 95\%$ that the Met Target sub group have a lower median target than the Target not Met sub group.

Remarks #

The obvious difference between groups is expected, as per the client brief (different customer segments), so the groups were considered separately here. A Bayesian t-test confirms the visual assessment that there is no significant difference in the median quotas set for the sub-groups in Group A (those who met their target vs those who missed it). One the other hand, I found evidence to suggest that this is not the case for Group B. Those who met their targets (taken as a group) may have had slightly more achievable targets than those who did not. Note that the evidence is borderline credible.

A standard t-test (not shown) produced a very similar result but the difference in Group B would be reported as non-significant based on the p-values (p = 0.056). Again, I might consider this a borderline result but the Bayesian interpretation is more intuitive.

A visual analysis of the MCMC diagnostics (not shown) revealed no problems with the test convergence.

Conclusion #

In light of this finding I suggested a review of the quota setting for group B, in order to ensure fairness across the team.

Appendix A: Descriptive Statistics #

Here are the standard summaries for the whole sample, and separately by group.

`1`	`## Both Groups`

##  Group     Attainment         KSales           KQuota       MetTarget      
##  A:346   Min.   : 55.35   Min.   : 6.282   Min.   : 6.813   Mode :logical  
##  B: 76   1st Qu.: 93.25   1st Qu.:20.279   1st Qu.:20.714   FALSE:233      
##          Median : 98.68   Median :28.271   Median :27.791   TRUE :189      
##          Mean   : 99.37   Mean   :30.140   Mean   :30.363                  
##          3rd Qu.:104.00   3rd Qu.:37.284   3rd Qu.:37.786                  
##          Max.   :150.12   Max.   :86.803   Max.   :82.044

`1`	`## Group A only`

##  Group     Attainment         KSales           KQuota       MetTarget      
##  A:346   Min.   : 73.73   Min.   : 8.969   Min.   : 9.873   Mode :logical  
##  B:  0   1st Qu.: 93.88   1st Qu.:23.605   1st Qu.:24.513   FALSE:190      
##          Median : 99.11   Median :30.601   Median :31.171   TRUE :156      
##          Mean   : 99.37   Mean   :32.949   Mean   :33.154                  
##          3rd Qu.:103.75   3rd Qu.:40.309   3rd Qu.:39.982                  
##          Max.   :139.73   Max.   :86.803   Max.   :82.044

`1`	`## Group B only`

##  Group    Attainment         KSales           KQuota       MetTarget      
##  A: 0   Min.   : 55.35   Min.   : 6.282   Min.   : 6.813   Mode :logical  
##  B:76   1st Qu.: 87.72   1st Qu.:13.770   1st Qu.:13.089   FALSE:43       
##         Median : 95.00   Median :16.462   Median :17.322   TRUE :33       
##         Mean   : 99.33   Mean   :17.353   Mean   :17.656                  
##         3rd Qu.:107.80   3rd Qu.:20.534   3rd Qu.:20.392                  
##         Max.   :150.12   Max.   :39.890   Max.   :43.988

Appendix B: Code #

library(readr)
library(dplyr)
library(lattice)
library(vcd)
library(BayesianFirstAid)
source("TC_Theme.R")
link <- "TC_SalesQuota.csv"
raw.data <- read_csv(link) %>%
  mutate(Group = as.factor(Group)
            , KSales = Sales/1000
            , KQuota = Quota/1000
            , Sales = as.integer(Sales)
            , Quota = as.integer(Quota)
            , MetTarget = Attainment >= 100)

get.col <- function(colnm, grp) {
  as.matrix(raw.data %>% filter(Group == grp) %>% dplyr::select(colnm))
}
set.seed(12321)

cat("Both Groups")
summary(k.data)
cat("Group A only")
summary(k.data %>% filter(Group == "A"))
cat("Group B only")
summary(k.data %>%  filter(Group == "B"))

mns <- round(tapply(raw.data$KQuota, list(raw.data$Group, raw.data$MetTarget), mean),2)
mns <- mns[, c("TRUE", "FALSE")]
bwplot(KQuota ~ factor(MetTarget, levels = c(TRUE, FALSE)) | Group
       , data = raw.data
       , scales = list(format = list(digits = 10))
       , par.settings = MyLatticeTheme
       , strip = MyLatticeStrip
       , xlab = "Met target"
       , ylab = "Quota (1000's units)"
       , main = "Group performance to targets"
       , panel = function(x, y, ...) {
         panel.bwplot(x, y, ...)
         panel.average(x, y
                       , lwd = 2
                       , lty = 1
                       , col = myPalDark[1]
                       , ...)
         panel.text(1:2
                    , mns[which.packet(),] + 5
                    , mns[which.packet(),]
                    , col = myPalDark[1]
                    , cex = 1.1
                    , fontface = "bold"
                    , ...)
       }
)

tta <- bayes.t.test(get.col("KQuota","A")[get.col("MetTarget", "A")]
            , get.col("KQuota","A")[!get.col("MetTarget", "A")]
            , var.equal = TRUE)

tta$x_name <- "Group A: Met Target"
tta$y_name <- "Group A: Target not Met"

plot(tta)

ttb <- bayes.t.test(get.col("KQuota","B")[get.col("MetTarget", "B")]
            , get.col("KQuota","B")[!get.col("MetTarget", "B")]
            , var.equal = TRUE)

ttb$x_name <- "Group B: Met Target"
ttb$y_name <- "Group B: Target not Met"

plot(ttb)

Be Inclusive, Be Change

Tue, 12 Jul 2016 00:00:00 +0000

Recently, I had the opportunity to work with B-Change.org (a.k.a. Be-Inclusive.org) to clarify and focus strategy and positioning for their mobile app. B-Change strives to help marginalized groups, to feel more connected. Their app was intended to be a relaunch of a community web-site that served the Southeast Asian LGBT and HIV-positive youth community by offering them information about service providers who are more sympathetic to their specific needs. The app helped B-Change to deliver a much more socially connected experience with a renewed focus on user genenarated content but they were struggling with user adoption and traction.

Their goal for engaging with me was to help them better reach their target audience through the app and improve the effectiveness of their platform. I designed and facilitated a couple of personalized, creative brainstorming and innovation sessions in Singapore and Sydney, using Innvation Games by Luke Hohmann. I weaved in ideas from the hooked model of app design by Nir Ayal. Through these sessions, we were able to identify key areas where the app could be improved and developed a roadmap for future updates and improvements.

It was an incredibly rewarding experience, and I’m proud to have been a part of such an important project. The work that Be-Change is doing is incredibly important. It’s inspiring to see organizations like this that are dedicated to making the world a more inclusive and welcoming place.