Why Gaussian Process Regression?

1. It’s a non-linear regression model
2. It also does built-in modeling of uncertainties (cool)

The following plot from the sci-kit learn documentation shows the output of GP regression model. Note the beautifully curved uncertainty bands. Where does all this come from?

GP Regression: The One Line Definition

Turns out GP regression can be described in one (somewhat loaded) line:

GP Regression: A Multivariate Gaussian Distribution over functions, conditioned on some training data.

Hmm. Maybe not so terrible? But how does this connect to the idea of “a nonlinear regression that models uncertainty”?

Let’s unpack this definition bit by bit, starting with “distribution over functions”.

Prequel: Distributions over Functions

We all know linear regression. Given some data, we assume it has the form . We have some data and , and we want to find the and (“^” means “estimated”) that describe a line of best fit. We can do this using OLS (ordinary least squares), and we end up with a linear function , that for any gives us a prediction for .

Note that we don’t model ; what if we want some notion of the uncertainty in our predictions?

For example, look at the two plots below:

Interactive figure: open the post to see it.

Even though the data on the left is much more noisy, both linear regressions model the data with the sample simple line. There’s no measure of uncertainty. Let’s try to add some.

What if (just for fun) we assumed and were actually random variables, and that they are normally distributed with variance equal to the variance in the data? Then our linear regression equation would look like this:

where we’ve converted our betas to random variables as follows:

This reads “ is a random variable distributed according to a Gaussian with mean and variance ”. We’ll set to be the variance of the data about the best fit line (the variance of the residuals). For the plots a and b above, then we’d have:

where and are gaussians with variance equal to the variance of the residuals in each case: and

Now that the betas are random variables, let’s sample from them, and plot the resulting lines:

(Click the “New Sample” button below)

Interactive figure: open the post to see it.

Each time you click the button, a new sample are drawn from the distributions of and (one set of betas for the noisy data on the left and one set for the less noisy on the right), and the corresponding lines are plotted.

Notice what we’ve created: it’s a distribution over functions. Each sample from is a different function . Also notice that as you sample more and more, the “spread” or uncertainty of the original fit becomes more and more apparent. You can also see that the uncertainty is higher in the noisier data.

This framework is just to develop for intuition for distributions over functions. For a more rigorous take, check out the bayesian linear regression section here.

Returning to our original definition, we’ve checked off one core idea:

GP Regression: A Multivariate Gaussian Distribution over functions, conditioned on some training data.

What if we want to model a non-linear relationship? Now we’re getting closer to the core idea of GPs. But of course, before we get to Gaussian processes, we have to talk about Gaussians.

Gaussians…and Multivariate Gaussians

The Gaussian, the bell curve, the normal distribution, we’ve seen it before. It’s defined by

where is the mean and is the variance. You can play around with the parameters of the distribution below. Each time you change the sliders, 10,000 samples are drawn from the distribution and plotted as a histogram.

Interactive figure: open the post to see it.

What about multivariate Gaussians? That is, a Gaussian distribution but with more than one variable. They are basically the same thing, but instead of having a single mean and variance, there’s there’s a mean vector and a covariance matrix.

So while a single Gaussian is this:

Where is the mean and is the variance, a multivariate Gaussian is this:

Where is a vector of means, is a covariance matrix.

In the above expressions, is the mean of the th component of

In this case when we sample from , we get a vector , instead of just a single value.

Nothing too crazy right? Well, except that the multivariate normal distribution has a covariance matrix, not just a variance.

In the covariance matrix we specify not only the variance of each variable, but also the covariance between each variable and every other. Each of the variables in a multivariate Gaussian can be correlated with each other. Below is an overhead view of a bunch of samples for a 2D multivariate Gaussian distribution. You can interactively change the covariance matrix to see how it affects the distribution.

Interactive figure: open the post to see it.

Note that if you change the diagonal elements, the variance of one or the other variable will change (look at the scale of axes). If you change the off-diagonal elements, the covariance between the two variables will change.

You also probably saw that the matrix has to be symmetric and with positive diagonals. Why? Well think about what covariance is: its how two variables vary together. It doesn’t make sense for Cov(X,Y) != Cov(Y,X). And it doesn’t make sense for Cov(X,X) — that is, Var(X) — to be negative (you’ll get an error if you try).

Also note that while the combined joint distribution changes form, each of the individual distributions is still a Gaussian. That is to say, the distribution of is still a gaussian, and the distribution of is still a Gaussian. The only thing that changes is the covariance between the two variables.

Gaussian Distribution Over Functions

So far we’ve visualized a multivariate Gaussian of just two variables, but you can imagine taking this into many many dimensions. That is to say, many many Gaussian random variables that may or may not be correlated with each other according to some big covariance matrix.

In order to think about more than two dimensions, we’ll need to visualize our distributions differently. Let’s start with a 1-D Gaussian. We’ll draw samples from the distribution and plot them below:

Interactive figure: open the post to see it.

Each new sample is drawn from a normal distribution with mean 0 and variance 1.

What if we want to sample from a multivariate normal distribution? Well, this time is a vector of random variables. If is 2-d then , so each sample is a vector of two values which we’ll plot each on their own part of the X-axis.

Interactive figure: open the post to see it.

So now each time we sample from a bivariate normal distribution, we get a vector of two numbers, which we plot as two connected points. Now let’s look at a 3-D Gaussian, ,

Interactive figure: open the post to see it.

Now let’s take it to an extreme: a 50-D Gaussian. , so every sample is a vector of 50 random values.

Interactive figure: open the post to see it.

Cool, so now we can visualize samples from a 50-D Gaussian in this kinda of weird way. How is this useful? And how does this relate to a Gaussian Process regression?

You’ll notice the above plot has a “Connect” button. If you click it, each variable in a given sample gets connected to the next.

Do these connected samples remind you of anything? Maybe a certain class of elementary mathematical objects?

If you thought “functions” then you are getting what I’m going for here. Each sample from the 50-D Gaussian is starting to look like some curve where, for any x-coordinate (which is just the index of the vector output of the multivariate gaussian) you can look up a y-value. So this kinda-sorta function can be defined as

given a multivariate Gaussian .

But remember what each different colored curve is: it’s a sample from a multivariate normal distribution. It’s almost like the 50-D Gaussian specifies a distribution over functions…

Almost. The 50-D Gaussian really only specifies a distribution over 50 discrete values, so it’s not quite a distribution over functions that can take any value of . Hold that thought for now, we’ll return to this later.

But first you might have some questions. Remember, the above plot was made by taking a few samples from 50-D gaussian,

But I didn’t tell you what and were. Well, I actually used a mean vector of all zeros:

and for the covariance matrix I simply used the Identity matrix.

Since this covariance matrix is really big, let’s visualize it with a heatmap:

Interactive figure: open the post to see it.

This means that each of the variables in our multivariate normal distribution are…..????

No variable has any covariance with any other, so you can think of this multivariate Gaussian as simply 50 separate Gaussians, each with mean = 0 and variance 1. This is why the curves we plotted above are so all over the place: each point is randomly bouncing up and down with no influence from its neighbors.

What if we add some non-zero values to the covariance matrix that are off-diagonal?

Remember the diagonal elements of the matrix at row and column is while the off-diagonal elements at are .

For example, check out the covariance matrix below:

Interactive figure: open the post to see it.

It looks very similar, but it’s “fuzzier” around the diagonal. Think about what type of values you would expect from a Gaussian with this covariance matrix.

(Try playing around with the mysterious “ℓ” slider, will explain later).

It’s saying that variables near each other are more correlated than variables far away. For example variable 1 is more correlated with variable 2 than it is with variable 50. Let visualize some samples from a 50-d Gaussian with this new covariance matrix. But before you hit the “sample” button, what do you think these new curves will look like?

Interactive figure: open the post to see it.

Cool, they are now much smoother. This is because the variables near each other are more correlated, so nearby points are more likely to be similar. This smooths out the curves.

Now we can see that by changing the covariance matrix, we can control the shape of the functions that our multivariate normal distribution produces.

# The RBF kernel, implemented in python with numpy
def pairwise_rbf(xa, xb, l=5.0):
    sq_norm = (-0.5 / l**2) * sp.spatial.distance.cdist(xa, xb, "sqeuclidean")
    return np.exp(sq_norm)

Conditioning on Data

Wow, we’ve come a long way. Let’s regroup here and remember our original definition of a Gaussian Process Regression.

GP Regression: A Multivariate Gaussian Distribution over functions, conditioned on some training data.

In fact this highlighted part above is the definition of a Gaussian Process. But we want “Gaussian Process Regression”. (People sometimes use the two to mean the same thing, but technically GP is the distribution and GPR is the regression problem.)

We’re more than half-way done! But we still need to condition it on some training data. Right now we’re just sampling these pretty functions, but they are completely random.

Conditioning means we want the probability of some outcome given some data. Mathematically we write this as:

that is, the probability of some outcome given some other data .

Let’s take a simple toy problem as a concrete example:

Say we are trying to predict the cost of a house along a particular road. At one end of the road, there is a nuclear power plant (yikes). We have some data on the cost of some of the houses on this road, but we want to predict the cost of a house at any location. Let’s look at the data first:

Interactive figure: open the post to see it.

Right away we see a couple things. Housing costs are not just linearly increasing as we get further away from the plant; there seem to be a few dips. Maybe there is a prison 5 miles away from the plant; who knows. Regardless, we probably want some non-linear model for this data.

The main question at hand is given this known cost data, what is the expected cost of a house at another location ?

To state this mathematically, we want some function structured as follows:

That is, we want the probability distribution of cost of a house , given its distance from the nuclear plant. Since we’ve been thinking about multivariate Gaussian, let’s make an assumption that the distribution above follows a multivariate Gaussian.

Where is the covariance function, and is the mean function. For now we’ll just assume that the mean function is zero, and for the covariance function we’ll use the RBF kernel.

Why these? Remember the RBF function’s effect above: it sort of “smooths” out the data. This makes sense to model housing prices, because it’s reasonable to expect that houses near each other will have similar prices.

But why is the mean function zero? A: trust me bro. It just works out in practice that mean = 0 is usually good enough to model the data. For example sklearn’s GP implementation doesn’t even let you specify a mean function. For more on this see Section 2.7 here.

We really need the following if we want to condition on the known data:

Putting this into words, we want the probability distribution of prices of a house, given its location and given the prices of houses at some other locations.

Let’s assume that the known data also came from the same multivariate Gaussian distribution as the unknown. So we can write:

And for the unknown data, remember we had:

Notice that we’ve assumed both the known data and the unknown data come from multivariate Gaussians. What if we just combined them into one big multivariate gaussian? We could basically just stack the distributions on top of each other to form a big “mother” Gaussian.

The one wrinkle here is the covariance function. For the off-diagonal elements, we need to use the covariance function between the unknown data and the known data. So now we see why the kernel function is written as ; in this case we need to compute the covariance between two different vectors of data to fill in the off-diagonal blocks of the covariance matrix above.

Can we massage the beast above into a probability distribution conditioned on the known data? Like so:

The answer is yes. It’s the following:

Please don’t get mad at me for just giving you the answer. It’s a kind of complicated derivation, and I don’t want us to get bogged down. If you want to go through it, see this section of Gaussian Processes for Machine Learning.

For now, just accept that there’s a nice closed form solution to this problem.

Wait, uh, so we’re done?

GP Regression: A Multivariate Gaussian Distribution over functions, conditioned on some training data.

We’re done! We’ve got a nice closed form distribution over functions conditioned on some data.

Actually Fitting a Regression Model

Using our conditional distribution above, we can plug in our known house costs to create a conditional distribution of functions. Then we can sample from this distribution at, say, 50 evenly spaced distance from 0 to 12 miles away from the nuclear plant.

We have 15 known cost data points. This means that and is are vectors of length 15. We want to predicted prices at 50 evenly spaced points, so and are vectors of length 50.

Plugging this into the conditional distribution above, we can now sample from the conditional distribution of functions:

Note: we’re using the RBF kernel here with parameter set to 1.0.

Interactive figure: open the post to see it.

Yeah baby. Now this looks like a Gaussian process regression. Clearly the samples from the distribution are conditioned on known data, because all the functions we sample pass through the known data points. But in between the known data points the functions are free to somewhat randomly vary, giving us an idea of the uncertainty. How smoothly the functions vary is determined by the covariance function, which in this case is the RBF kernel.

In the plot below, I reveal the true underlying function I used to generate this “housing data” (pink). What if we take 500 samples from this posterior distribution?

Interactive figure: open the post to see it.

Remember what there blue lines are: they are samples from a multivariate gaussian conditioned on the known data points, and sampled at 500 evenly spaced points between 0 and 12. We used the complicated formula above to find the conditional mean vector and conditional covariance matrix, and then sampled from a distribution using that mean and covariance.

Below are some heatmaps of the conditional covariance matrix and conditional mean vector (conditioned on the known housing data) that specify the predictive multivariate gaussian distribution.

Think for a moment about why this conditional mean and covariance makes sense.

Interactive figure: open the post to see it.

Outro

You should now have the core ideas of Gaussian processes regression. If you still have some questions, like:

• “Do you really have to sample 100s of functions to get the confidence intervals?”
  • (You should now be able to figure out the answer to this.)
• “How do we choose a kernel function?”
• “How do we choose the best parameters for the kernel function?”
• “What if the training data is intrinsically noisy (the price of houses has some variance at a given location)”
• “What if there are many features in my training data? E.g. Square footage of houses.”
• “But I heard GPs are expensive to train?”

…then you should check out the resources below, which should be easier to understand now that you have the basics.

References

Level 2: Rust

I realized this would be a perfect opportunity to try out Rust, which is supposed to make writing extremely fast code safer and easier.

// rust hand-rolled
#[pyfunction]
fn transcribe(dna: &str) -> String {
    dna
        .chars()
        .map(|x| if x == 'T' { 'U' } else { x })
        .collect()
}

# calling rust functions from python
import bio_lib_string_rs

dna = "ACTGACTC"
bio_lib_string_rs.transcribe(dna)

# python hand-rolled 
def transcribe(dna: str) -> str:
   return ''.join(["U" if char == 'T' else char for char in dna])

# python built-in
def transcribe_builtin(dna: str) -> str:
    return dna.replace("T", "U")

// rust built-in
#[pyfunction]
fn transcribe_builtin(dna: &str) -> String {
    dna.replace("T", "U")
}

# numpy 
import numpy as np

def transcribe_np(dna: str) -> str:
    return str(np.char.replace(dna, "T", "U"))  # type: ignore

@dataclass
class PalindromeLocation:
    start_index: int
    length: int


def find_reverse_palindromes(seq: str) -> List[PalindromeLocation]:
    min_len = 4
    max_len = 12
    locations = []
    for i in range(0, len(seq) - min_len + 1):
        for length in range(min_len, max_len + 1, 2):
            if i + length > len(seq):
                continue
            test_seq = seq[i:(i + length)]
            if is_reverse_palindrome(test_seq):
                locations.append(
                    PalindromeLocation(start_index=i + 1, length=length)
                )
    return locations


def is_reverse_palindrome(seq: str) -> bool:
    return seq == reverse_complement_dna(seq)


def reverse_complement_dna(dna_seq: str) -> str:
    return ''.join([dna_base_complement(b) for b in dna_seq[::-1]])


def dna_base_complement(base: str) -> str: 
    if base == "A":
        return "T"
    elif base == "T":
        return "A"
    elif base == "G":
        return "C"
    elif base == "C":
        return "G"
    else:
        raise Exception("Non-DNA base \"{}\" found.".format(base))

#[pyclass]
pub struct PalindromeLocation {
    #[pyo3(get, set)]
    pub start_index: usize,
    #[pyo3(get, set)]
    pub length: usize,
}

#[pyfunction]
pub fn find_reverse_palindromes(seq: &str) -> Vec<PalindromeLocation> {
    let min_len = 4;
    let max_len = 12;
    seq.chars()
        .take(seq.len() - min_len + 1)
        .enumerate()
        .fold(Vec::new(), |mut acc, (i, _)| {
            for length in (min_len..(max_len + 1)).step_by(2) {
                if i + length > seq.len() {
                    continue;
                }
                let test_seq = &seq[i..(i + length)];
                if is_reverse_palindrome(test_seq) {
                    acc.push(PalindromeLocation {
                        start_index: i + 1,
                        length,
                    });
                }
            }
            acc
        })
}

pub fn is_reverse_palindrome(seq: &str) -> bool {
    seq == reverse_complement_dna(seq)
}

pub fn reverse_complement_dna(dna_seq: &str) -> String {
    dna_seq.chars().rev().map(dna_base_complement).collect()
}

pub fn dna_base_complement(base: char) -> char {
    match base {
        'A' => 'T',
        'T' => 'A',
        'G' => 'C',
        'C' => 'G',
        _ => panic!("Non-DNA base \"{}\" found.", base),
    }
}

def find_reverse_palindromes_rs(seq: str) -> List[PalindromeLocation]:
    ps = bio_lib_string_rs.find_reverse_palindromes(seq)
    return [ 
        PalindromeLocation(
            start_index=p.start_index,
            length=p.length
        )
        for p in ps 
    ]

Level 3: Algebraic Data Types

So far I’ve just represented DNA as strings. This is bad, and here’s an example of why.

pub fn dna_base_complement(base: char) -> char {
    match base {
        'A' => 'T',
        'T' => 'A',
        'G' => 'C',
        'C' => 'G',
        // Commented out for now...
        // _ => panic!("Non-DNA base \"{}\" found.", base),
    }
}

This Rust little function, when given a DNA base, returns the complementary base. Note I had to name it dna_base_complement, because it only works with DNA. If you understand basic biology, you know the output is also DNA (not RNA, or amino acids). However, none of this information is encoded in the logic of the function’s code.

I’ve commented out a line in the code above, and without it, the function actually doesn’t compile. Rust checks pattern matches for exhaustivity, and since any UTF-8 char can be passed into this function, I have to also handle the case where the base argument happens to not be “A”, “C”, “T”, or “G”.

So I have to uncomment that last line, which is a catch-all case. Now if I somehow give a non-DNA character to our function at runtime, the program will panic (and crash if the panic isn’t handled). Say I accidentally pass in the RNA character “U”…uh oh! Thankfully I can completely eliminate the possibility of this kind of runtime error using Algebraic Data Types or ADTs.

Algebraic data types are simply types composed of other types. There are two main kinds of ADTs: product types and sum types. A product type is an AND group of types: for example tuples, structs, or Python dataclasses. These are pretty obviously useful: sometimes you need to group diversely typed data together under one type, like a user type that has string name AND integer age fields.

The other common ADT, the sum type, was new to me, but I’ve realized it’s perhaps even more powerful and interesting⁹. A sum type is an XOR group of different types, so an instance can be one (and only one) type out of a set of given options. In Rust you create sum types with the enum keyword. For example:

enum DnaNucleotide {
    A,
    C,
    G,
    T,
}

This defines DnaNucleotide as a new type that can be one of four variants¹⁰ A, C, G, or T. Why is this interesting? Well, now I can rewrite my complement function like so:

fn complement(base: DnaNucleotide) -> DnaNucleotide {
    match base {
          DnaNucleotide::A => DnaNucleotide::T,
          DnaNucleotide::T => DnaNucleotide::A,
          DnaNucleotide::C => DnaNucleotide::G,
          DnaNucleotide::G => DnaNucleotide::C,
    }
}

Note I dropped the dna_ prefix from the function name: I know I’m getting the complement of DNA because the base argument is of DnaNucleotide type, and so is the return type. It only accepts DNA, not RNA or “!” or “为” or “🌯”. And if I try to pass in a character I get a compile time error:

We aren’t even allowed to wire complement up to anything but its proper DNA input. We also get another neat exhaustivity check at compile time if we forget to handle one of the enumerated bases:

This time I don’t have to add the catch-all _ => panic!("Non-DNA base \"{}\" found.", base) case because the compiler knows there can only be four different DnaNucleotide variants, and I’ve properly handled all of them. While this example might seem trivial, what about, say, a function from codons to amino acids? If I forget or duplicate one of the codons (like I did below), the compiler tells us!

Note in this example I’ve made use of RnaNucleotide and AminoAcid enums that I defined elsewhere in the code.

This is also useful for easily adapting the software to work with alloproteins (proteins with non-natural amino acids) or artificial base pairs. All I’d have to do is add another symbol to the “AminoAcid” or “DnaNucleotide” enums, and then a bunch of exhaustivity checking compiler errors will pop up wherever I now need to handle the new variant types.

#[pyfunction]
pub fn find_reverse_palindrome_dna(seq: &str) -> Vec<PalindromeLocation> {
    let seq = DNA::parse_string(seq);
    find_reverse_palindromes(&seq)
}

Level 4: Parallelize

So far I’ve been ignoring one of the most important speed factors in modern programming: parallelism. It’s always slightly painful to see my six-core Intel i7 running at, well, 1/6 capacity!

I decided to try out the newish ray package to parallelize my Python function¹¹. The code for finding reverse palindromes ended up being reasonably similar to the single-threaded version, though it requires an extra BATCH_SIZE parameter which needs be tuned to optimally slice up work into chunks:

import ray
import functools
import operator


def find_reverse_palindromes_par(seq: str) -> List[PalindromeLocation]:
    BATCH_SIZE = 100
    min_len = 4
    max_len = 12
    locations = []
    ray_seq = ray.put(seq)  # type: ignore

    @ray.remote  # type: ignore
    def is_palindrome(i: int) -> List[PalindromeLocation]:
        seq = ray.get(ray_seq)  # type: ignore
        locations_inner = []
        for i in range(i, i+BATCH_SIZE):
            for length in range(min_len, max_len + 1, 2):
                if i + length > len(seq):  # type: ignore
                    continue
                test_seq = seq[i:(i + length)]  # type: ignore
                if is_reverse_palindrome(test_seq):  # type: ignore
                    locations_inner.append(
                        PalindromeLocation(start_index=i + 1, length=length)
                    )
        return locations_inner

    for i in range(0, len(seq) - min_len + 1, BATCH_SIZE):
        locations.append(is_palindrome.remote(i))
    return functools.reduce( # flatten list of list
        operator.iconcat, ray.get(locations), []  # type: ignore
    )

Now let’s run this and take a look at our CPU monitor:

Yeah baby! All six cores fully engaged sir!

Parallelizing the Rust code turned out to be the first case where implementation is actually easier in Rust than in Python thanks to the excellent rayon package:

pub fn find_reverse_palindromes_par(seq: &DNASlice) -> Vec<PalindromeLocation> {
    let min_len = 4;
    let max_len = 12;
    seq.into_par_iter()
        .take(seq.len() - min_len + 1)
        .enumerate()
        .fold(Vec::new, |mut acc, (i, _)| {
            for length in (min_len..(max_len + 1)).step_by(2) {
                if i + length > seq.len() {
                    continue;
                }
                let test_seq = &seq[i..(i + length)];
                if is_reverse_palindrome(test_seq) {
                    acc.push(PalindromeLocation {
                        start_index: i + 1,
                        length,
                    });
                }
            }
            acc
        })
        .reduce(
            Vec::new,
            |a: Vec<PalindromeLocation>, b: Vec<PalindromeLocation>| [a, b].concat(),
        )
}

It’s as simple as changing .iter to .into_par_iter and adding a .reduce function at the end to stitch together all the asynchronously returned results. Of course, it also needed to be wrapped in similar Python→string→enum/ADT wrapper functions to be useable from Python.

So now it’s time for the final showdown. How do all these “levels” compare speed-wise?

Remember these results are all calls from Python.

While both parallel implementations are slower for smaller inputs (as expected), the Python version is much slower, and it eventually returns to being slower than sequential Python. This was because the BATCH_SIZE parameter needed some tuning, and after some tedious brute force experimentation, I ended up finding a slightly better value (that was still slower than single-threaded Rust). However Rust’s rayon performed excellently out of the box thanks to its built-in dynamic performance tuning.

It’s neat to see the performance gains for each of our “levels” of code improvement. In the end, I made my code safer, better organized, and faster by nearly two orders of magnitude. The overhead of learning Rust is certainly high, but hey, I’m already over that hump!

Level 5?

To avoid publication bias, I admit I also tried a “Level 5” improvement by using iterators more heavily. But, well, gather round…

I was discussing this project with a hacker friend who suggested modifying my functions to return iterators instead of vectors. This way, I could chain together various transformation functions lazily and only call .collect when needed. The compiler could then optimize the entire chain of transformations top to bottom instead of being forced to collect into vector at each step. Since I’m a data scientist with a Spark background, this suggestion made a lot of sense.

This is where implementing things in Rust got incredibly tricky, and frankly very unproductive. I had to switch over to Rust’s nightly build to be able to make use of experimental typing features. I had to rewrite one line functions as 25-line home-made iterator implementations. I had to really get in the mud with lifetimes, traits, and generics, leading to function signatures like:

pub fn transcribe<'a>(seq: DnaIter<'a>) -> impl Iterator<Item = RnaNucleotide> + 'a
where
    DnaIter<'a>: 'a,
{
    seq.map(|b| transcribe_base(&b))
}

In the end, my code was much less clean, though I could have possibly made it cleaner by using performance-sapping dynamic dispatch¹².

I should have stopped and benchmarked sooner, but instead I struggled to port the entire set of bioinformatics functions I’d written so far to this iterators based implementation. The entire monstrosity is available here. And yes, in the end, the code was significantly (~10%) slower. See one benchmark below:

It would have needed to be a lot faster to justify all the extra work and codebase messiness. Given it was pretty hard to find examples of people using this iterator pattern online, I’m guessing it’s pretty much an anti-pattern. The Rust compiler is clearly worse at optimizing code written this way. In the Rust community, there’s a lot of talk about “idiomatic” code, and I see the value. Unless you’re fluent in assembly and compilers (I’m not), just writing Rust idiomatically seems like an important heuristic for also getting performant Rust. In this case, I violated a universal idiom of all programming languages: “if you’re writing lots of complicated code to do simple things, you’re probably doing it wrong.”

I guess 4 levels of improvement was enough…

TL;DR

• Rust is faster than Python, but not necessarily for very simple things.
• It’s pretty easy to incrementally add fast Rust functions to an existing Python codebase.
• Use algebraic data types like enums instead of strings. ADT-based code is cleaner, safer, and faster.
• Parallelization is important, and it’s easy(er) to do in Rust.
• Benchmark, don’t theorize. “Humans are terrible at guessing about performance!”
• Writing Rust simply and idiomatically will probably result in the fastest code (as well as the cleanest).

Special thanks to @jgavris of rs-git-fsmonitor fame for various tips and pointers on this post.

Notes

Straight Ticket Voting: An Electoral Oddity

Ayyadurai’s analysis centers around an odd aspect of Michigan voting: straight ticket (or straight party) voting. In a few states, Michigan one of them, you can skip voting for individual candidates for president, senate, congress, and other downballot offices. Instead, you can just check one box to vote for “The Democratic Party” (or “The Republican Party”), which automatically casts your votes to the Democrat (or Republican) candidates for every office on the ballot. It’s basically a “select all” option.

Ayyadurai’s team created a data visualization by plotting the difference between the straight-ticket voting compared with the regular candidate-by-candidate voting (which I’ll refer to as “mixed-ticket”, since you can mix which parties you vote for when voting in this way). First, they organized precinct voting data as follows:

For each precinct in a county, they show (2nd column from the left) what percentage of straight ticket votes went to Trump. Then they calculate how the overall vote went (3rd column from left), and finally look at the difference between the two. They visualize this data with a plot like so (here showing the example data from above):

Their thesis is that the straight-ticket voting patterns should be similar to the overall voting patterns. So in a given precinct, if 60% of the straight-ticket votes were for Trump, Ayyadurai pretty reasonably theorizes that around 60% of the mixed party votes should also go to Trump, giving us a flat horizontal line in our plot.

However, when Ayyadurai looked at the actual data from several Michigan counties, he found something much different: a steep and highly significant downward trend, implying that the more Republican a precinct is, the more it tended to flip toward Biden.

It looks like his votes counts start getting linearly reduced and if you look at this curve and I’ve looked at lots of curves…anyone of us who’s in the pattern analysis business when you see this…you say something’s up. This is too structured, right? Too perfect.

He’s right: this level of trend is something that anyone who’s done data science usually loves to see: a very (very) strong correlation between two variables.

What’s going on? This puzzled me too at first. Perhaps it’s caused by “Romney Republicans” that don’t like Trump? Well, Ayyadurai has an answer to that too: if Republicans showed a distaste for Trump, we still ought to see a flat chart, just shifted down. Something else is going on, and for Ayyadurai, the only option left to consider is a malicious algorithm.

And remember we’ve talked about it the election systems…they have the feature of being able to manipulate vote counts from one person to another. It’s called a weighted race feature.

Why Are The Most Republican Precincts Flipping Toward Biden?

That problem is, he’s gotten slightly confused about what he’s looking at. We are NOT actually comparing precincts by “how Republican they are”. We are comparing straight-ticket results to overall results. This is a small but very important difference. Why? When someone goes to the polls in Michigan, they first choose to either vote straight-ticket or mixed-ticket. And they don’t pick randomly. This is what’s called a selection bias in data analysis.

The voters that show up to the polls and check the “all Democrats” or “all Republicans” box are, by definition, more partisan voters than the voters that decide to pick and choose between both parties. If a precinct’s straight-ticket votes (the votes from the most partisan voters in the area) go very Trump, then yes, you would expect the definitionally less partisan voters in the area (the mixed-ticket voters) to vote less strongly that way.

But won’t some Biden supporters switch to Trump too? Yes, but not as many, because in a very Republican area Trump would have more to lose and less to gain. Let’s go through a quick example.

Say a precinct has 1000 constituents: 800 Republican, 200 Democrat. If 10% of the Republicans decide to flip, Trump loses 80 votes. If 10% of the Democrats decide to flip, Trump gains 20 votes, for a net loss of 60 votes. Given the selection bias at work, most of these flipping voters are going to do a mixed-ticket vote…if you are a Republican that hates Trump, you probably still want to vote GOP for Senate, etc.

Ayyadurai would say here, 60 votes got moved to Biden. But it’s actually just that mixed-ticket voters are less likely (by definition) to vote with precinct norms, and will tend to move overall results back towards the center.

All of this works the same way for more Democratic precincts; just do the same math but switch “Republican” and “Democrat”, and “Trump” and “Biden”. And if a precinct is split 50/50, there flipping votes net to zero. How does all this look on the precinct plots we’ve been looking at?

It’s a nice downward sloping line. I’ve only done three data points, but you can see how the rest of the line fills in depending on how Republican or Democrat a precinct is. You can get a steeper line by increasing the percentage of mixed voters that flip (I used a rate of 20% in the example above), or shift the line up or down by making the flip percentage assymentical (e.g. 20% of Republicans flip, but only 10% of Democrats do). Add in some random variation, and this is exactly what we’re seeing in Dr. Shiva’s Michigan chart. It’s theoretically totally normal².

Do Real Elections Actually Work This Way?

So we’ve got ourselves a theory about why these downward trends might exist, no fraud required. Let’s now do what Dr. Shiva didn’t do and look at more data. If our theory is right, we’d expect most counties to show this downward trend, and if Dr. Shiva’s is right, this downward trend should almost never appear.

Remember, only a few states do straight-ticket voting, and it turned out to be either quite a pain or impossible to get at their data. Ok.gov didn’t seem to work at all, much less have a detailed voting records API. Yikes.

Of course, we do know Michigan’s data is available, so lets take the first minimal step of verification Dr. Shiva didn’t bother to take: look at the same “suspicious” counties in the 2016 election, when Trump won Michigan. Is 2020 really an anomaly? Let’s look at Oakland county in 2016:

It’s the same super statistically significant downward linear trend. It’s not as steep as in 2020, but still equally “suspicious” according to Ayyadurai’s theory. What about the following counties:

Do these charts get your pattern researcher taste-buds salivating? Ready to take to the Twitters and yell fraud? Before you do, I should say this is data from Randoph, Jackson, Walker, Autauga, Geneva, and Fayette counties…in Alabama (2016 election)³. This is obviously not the first state that comes to mind when you think of election hacking targets; the state hasn’t voted for a Democrat president since Jimmy Carter in ’76.

I admit, I cherry-picked some of the “worst” counties above, but the same trend holds throughout. Running Ayyadurai’s analysis on all Alabama counties in 2016, 70% of them showed a negative, statistically significant slope (in other words β < 0 and p < 0.05). In fact, only 13 of the 67 counties were “normal” by Dr. Shiva’s definition, showing no significant trend either way.

So…Now What?

Now we are at a crossroads. We’ve got a theory for why such patterns could exist completely legitimately, and the data shows these “suspicious” patterns seem to be the norm.

I’d go with “there was no fraud” (at least of this kind). Dr. Shiva essentially cooked up a graph of “do mixed-party voters vote less partisan than fully partisan voters?” It’s almost a tautology, and if considered carefully, you’d expect a trend. I don’t know if my explanation is 100% correct, but it’s certainly much better supported by the data than Dr. Shiva’s theory that “trends are anomalous and signify fraud.”

Is Dr. Shiva lying to us then? Probably not. The selection bias he overlooked is easy to miss, and even the best data gurus out there make mistakes in identifying them.

Was Dr. Shiva doing good science and just happened to mess up? No, absolutely not. He and his team didn’t do even the most basic checks to support their theory, i.e. the checks I did above with 2016 election data (by myself in less than a day’s work). Even so, they decided to go ahead and claim there was a clear sign of fraud in Michigan, and publish this to their 200k trusting subscribers. Nothing could be more scientifically irresponsible, especially for a man that claims over and over again to be following the principles of science. Now I’m starting to wonder…did he really invent email?

Code and data used to generate this analysis can be found on Github.