Dimensionality reduction · R · Python

Supervised UMAP with Random Forests in Python and R

Exploring forest leaf embeddings with UMAP across Python and R, including corrected AUC evaluation, clustering code and the limits of the original comparison.

Recovered from the original publication. Historical tools and examples are preserved in context; this is not a claim that their integrations still run unchanged.

I wanted to bring a forest-based dimensionality reduction workflow from Python into R. The idea was to train a forest, extract the leaf each observation reached in every tree, and use those assignments as the input to UMAP. I followed Guilherme Marmerola’s walkthrough, using a smaller simulated dataset and comparing ways to extract that information from R packages.

The practical constraint was a Java-facing analysis interface for clinicians: people with expertise in the biology, but not necessarily in the computational methods. The analysis server provided one core, so compute time mattered alongside access to the leaf assignments.

Editorial update — November 3, 2026: This article has been edited from the original slideshow. The code below corrects evaluation and plotting mistakes. Historical outputs are identified separately; they are not results from a complete rerun of the corrected Python/R workflow. The original slideshow remains available as a record of the experiment.

What the forest contributes

Each row of the leaf-assignment matrix represents an observation; each column represents a tree. Two observations are similar when they repeatedly arrive at the same leaf within the same tree. The numerical size of a leaf ID has no meaning as a prediction score.

UMAP can use Hamming distance on these rows: the fraction of trees in which two observations land in different leaves. The supervision enters through training the forest on class labels. This is different from passing labels directly to UMAP’s y argument.

The original goal was to obtain these assignments for an R implementation such as uwot. For this comparison, I kept the embedding and plotting in Python and passed the R leaf matrices back through reticulate. That makes it easier to see which part of the workflow changes between forest implementations.

What the original run recorded

The simulated dataset contained 10,000 observations, 500 features and two classes, with ten generating clusters per class. The preserved console output reports:

Forest Recorded test ROC AUC Recorded clusters after cutting the sampled embedding
Python ExtraTreesClassifier 0.8504 19
R randomForest 0.8919 14
R ranger No valid probability-based AUC was recorded 17

The earlier prose incorrectly gave 12 clusters for randomForest and 14 for ranger. The table follows the console output. The ranger output also contained 0.5741, mistyped as 0.5471 in the prose, but fixing that typo would not fix the analysis: it was calculated from terminal-node IDs rather than class probabilities. It cannot support a claim about the probability forest’s classification performance.

These are exploratory results, not a ranking of the packages. The models used different settings, the R extraTrees fit used 500 trees while the other examples used 100, and the Python forest requested all available cores despite the intended single-core constraint. The saved timing comments also disagreed with the console output. There is no controlled speed comparison here.

Nor does getting close to 20 clusters establish that the underlying structure was recovered. The count depends on the embedding, sampling, linkage method and cut height. The original UMAP calls did not fix a random seed; the corrected examples do. Expect new runs to produce different coordinates and potentially different counts.

Set up the shared data

Run the Python blocks in the same Python session, through reticulate::repl_python() when working from R. Use exit to return to R, and select the intended Python environment with reticulate before starting it. The examples use NumPy, pandas, scikit-learn, UMAP, matplotlib, fastcluster and PyNNDescent; the R sections use reticulate, randomForest, ranger, pROC and, for the final experiment, extraTrees.

The original environment used Python 3.6. Treat it as historical context rather than a version recommendation. A fresh reproduction should record Python, R and package versions as well as the seeds and thread settings. See UMAP’s reproducibility notes.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from umap import UMAP
from fastcluster import single
from scipy.cluster.hierarchy import cut_tree
from pynndescent import NNDescent

plt.style.use('bmh')

X_raw, y = make_classification(
    n_samples=10000,
    n_features=500,
    n_informative=5,
    n_redundant=0,
    n_clusters_per_class=10,
    weights=[0.80],
    flip_y=0.05,
    class_sep=3.5,
    random_state=42,
)

# Share exactly the same split with R.
X_train_raw, X_test_raw, y_train, y_test = train_test_split(
    X_raw, y, test_size=0.3, random_state=42
)

# Fit preprocessing on the training partition, not the test data.
scaler = StandardScaler().fit(X_train_raw)
X_train = pd.DataFrame(scaler.transform(X_train_raw))
X_test = pd.DataFrame(scaler.transform(X_test_raw))
X = pd.DataFrame(scaler.transform(X_raw))

The original walkthrough used cross-validation. Here I used one train/test split so Python and R would evaluate the same observations. A shared split is useful, but it does not make different forest algorithms or hyperparameters equivalent.

Start with Python’s ExtraTreesClassifier

First evaluate the classifier on the held-out observations. Use the probability of class 1, not the predicted class or a terminal-node identifier.

et = ExtraTreesClassifier(
    n_estimators=100,
    min_samples_leaf=500,
    max_features=0.80,
    bootstrap=True,
    class_weight='balanced',
    n_jobs=1,
    random_state=42,
)
et.fit(X_train, y_train)
positive_column = np.flatnonzero(et.classes_ == 1).item()
probability = et.predict_proba(X_test)[:, positive_column]
print('Test ROC AUC:', roc_auc_score(y_test, probability))

The scikit-learn API exposes probabilities through predict_proba() and leaf assignments through apply(). They serve different purposes.

For the exploratory visualization, refit on all observations and extract the leaves. With 10,000 observations and 100 trees, the matrix has shape (10000, 100).

et.fit(X, y)
leaves_et = et.apply(X)
print('Leaf matrix shape:', leaves_et.shape)

sup_embed_et = UMAP(
    metric='hamming', random_state=42, n_jobs=1
).fit_transform(leaves_et)

This full-data refit uses all class labels. Its visualization is an exploration of the labelled dataset, not an independent test of how well the embedding generalizes to new observations.

A small plotting function keeps the later comparisons consistent:

def plot_embedding(embedding, labels, title):
    fig, ax = plt.subplots(figsize=(12, 7), dpi=150)
    for label in np.unique(labels):
        mask = labels == label
        ax.scatter(
            embedding[mask, 0], embedding[mask, 1],
            s=1, label=f'y={label}',
        )
    ax.set(title=title, xlabel='UMAP 1', ylabel='UMAP 2')
    ax.legend(fontsize=12, markerscale=5)
    plt.show()

plot_embedding(sup_embed_et, y, 'Forest-based embedding: ExtraTrees')

Cluster the embedding

The original approach takes 5,000 embedded observations, applies single-linkage hierarchical clustering, and cuts the tree at a height of 0.35. It then assigns the remaining observations the label of their nearest sampled neighbour in that same embedding.

The important detail is to use the same model’s embedding at every step. The original R sections sampled their own embeddings but queried and plotted sup_embed_et, the Python ExtraTrees result. The helper below takes the embedding explicitly to avoid that mix-up. It also preserves the sampled observations’ own cluster labels rather than querying them again.

def cluster_embedding(embedding, sample_size=5000, cut_height=0.35):
    sample = pd.DataFrame(embedding).sample(
        n=min(sample_size, len(embedding)), random_state=42
    )
    linkage = single(sample.to_numpy())
    sampled_labels = cut_tree(linkage, height=cut_height).ravel()

    index = NNDescent(
        sample.to_numpy(), n_neighbors=10, random_state=42, n_jobs=1
    )
    neighbours, _ = index.query(embedding, k=1)
    labels = sampled_labels[neighbours[:, 0]]
    labels[sample.index.to_numpy()] = sampled_labels
    return sampled_labels, labels

sampled_et, clusters_et = cluster_embedding(sup_embed_et)
print('Clusters in sampled embedding:', np.unique(sampled_et).size)


def plot_clusters(embedding, labels, title):
    fig, ax = plt.subplots(figsize=(12, 7), dpi=150)
    ax.scatter(embedding[:, 0], embedding[:, 1], s=1, c=labels, cmap='plasma')
    ax.set(title=title, xlabel='UMAP 1', ylabel='UMAP 2')
    plt.show()

plot_clusters(sup_embed_et, clusters_et, 'Clustering: ExtraTrees')

The original transcript repeated the same images under all three models, including a figure titled for ranger in the Python section. Those figures have been omitted because their placement does not reliably identify the run they depict.

Move the forest training into R

Back in R, copy the shared data and labels from Python. Make the class levels explicit so that a probability column named "1" always refers to the positive class.

library(reticulate)
library(randomForest)
library(ranger)
library(pROC)

df <- data.frame(py$X)
df$labels <- factor(py$y, levels = c(0, 1))

d_train <- data.frame(py$X_train)
d_test <- data.frame(py$X_test)
d_train$labels <- factor(py$y_train, levels = c(0, 1))
d_test$labels <- factor(py$y_test, levels = c(0, 1))
xcols <- setdiff(names(d_train), "labels")

randomForest

Fit 100 trees, then evaluate class probabilities on the same test partition. The ROC direction is explicit: larger scores mean stronger evidence for class 1. This avoids allowing pROC to choose the direction from the observed scores; see its ROC documentation.

set.seed(123)
rf_model <- randomForest::randomForest(
    x = d_train[, xcols],
    y = d_train$labels,
    ntree = 100
)
rf_probability <- predict(
    rf_model, newdata = d_test[, xcols], type = "prob"
)[, "1"]
rf_roc <- pROC::roc(
    response = d_test$labels,
    predictor = rf_probability,
    levels = c("0", "1"),
    direction = "<",
    quiet = TRUE
)
pROC::auc(rf_roc)

# Refit for exploration and retain the terminal node in every tree.
set.seed(123)
rf_full <- randomForest::randomForest(
    labels ~ ., data = df, ntree = 100, keep.forest = TRUE
)
rf_predictions <- predict(
    rf_full, newdata = df[, xcols], type = "prob", nodes = TRUE
)
leaves_rf <- attr(rf_predictions, "nodes")
dim(leaves_rf)

Return to the existing Python session with reticulate::repl_python(). The r object gives access to the R leaf matrix; the helpers defined earlier keep the plots and cluster assignment tied to this model.

leaves_rf = np.asarray(r.leaves_rf)
sup_embed_rf = UMAP(
    metric='hamming', random_state=42, n_jobs=1
).fit_transform(leaves_rf)
plot_embedding(sup_embed_rf, y, 'Forest-based embedding: randomForest')
sampled_rf, clusters_rf = cluster_embedding(sup_embed_rf)
print('Clusters in sampled embedding:', np.unique(sampled_rf).size)
plot_clusters(sup_embed_rf, clusters_rf, 'Clustering: randomForest')

ranger: probabilities for evaluation, leaves for embedding

The error in this part of the original experiment was the prediction type. With type = "terminalNodes", the second column contains IDs from the second tree. It is not the probability of the second class.

For a probability forest, ranger’s type = "response" returns class probabilities. Use that for ROC AUC; request terminal nodes separately when building the embedding. The corrected probability-based AUC still needs to be measured in a fresh run, so no replacement score is claimed here.

set.seed(123)
ranger_model <- ranger::ranger(
    labels ~ .,
    data = d_train,
    num.trees = 100,
    num.threads = 1,
    probability = TRUE
)
ranger_probability <- predict(
    ranger_model,
    data = d_test[, xcols],
    type = "response",
    num.threads = 1
)$predictions[, "1"]
ranger_roc <- pROC::roc(
    response = d_test$labels,
    predictor = ranger_probability,
    levels = c("0", "1"),
    direction = "<",
    quiet = TRUE
)
pROC::auc(ranger_roc)

set.seed(123)
ranger_full <- ranger::ranger(
    labels ~ ., data = df, num.trees = 100,
    probability = TRUE, num.threads = 1
)
leaves_rg <- predict(
    ranger_full, data = df[, xcols],
    type = "terminalNodes", num.threads = 1
)$predictions

Back in Python:

leaves_rg = np.asarray(r.leaves_rg)
sup_embed_rg = UMAP(
    metric='hamming', random_state=42, n_jobs=1
).fit_transform(leaves_rg)
plot_embedding(sup_embed_rg, y, 'Forest-based embedding: ranger')
sampled_rg, clusters_rg = cluster_embedding(sup_embed_rg)
print('Clusters in sampled embedding:', np.unique(sampled_rg).size)
plot_clusters(sup_embed_rg, clusters_rg, 'Clustering: ranger')

The R extraTrees experiment

I also tried R’s extraTrees package as another route to an extremely randomized forest. The original fit used 500 trees:

library(extraTrees)
set.seed(123)
model_extraTrees <- extraTrees::extraTrees(
    x = d_train[, xcols],
    y = d_train$labels,
    ntree = 500,
    numThreads = 1
)

At the time, I could not obtain the leaf indices needed for this workflow and made a feature request. The original experiment stops there for this package: it records a fit, but no corresponding embedding or test AUC. That historical limitation is not a claim about what the current package can do.

What this comparison establishes

The useful connection is between a supervised forest and a similarity representation: leaf assignments give UMAP something different from the original feature values. Python’s ExtraTreesClassifier, R’s randomForest and ranger expose those assignments, but evaluation still needs class scores rather than node IDs.

The original run explored that connection across languages. It did not settle which package was fastest or most accurate. A stronger comparison would match the model settings where possible, control threads, repeat the evaluation across splits and seeds, and regenerate each model’s figures. Until then, the historical results are context for the experiment, and the corrected code is a starting point for reproducing it.


← Back to Field Notes