Where does Spotify get that color from?

Over the years, I’ve done several “little things” that have had something to do, one way or another, with Spotify. I used the API to classify the music I listen to most by decade to see how old my tastes were, to connect it with Home Assistant, to separate music by language, and a few other bits. But in recent days I’ve been busy getting a clock to tell me what song I’m listening to.

That was where I built, for the first time, a graphical interface for my music project. This interface needed a colorful background for the image indicating which song was playing.
The problem is that, looking at the Spotify developer portal documentation, there was no reference to colors anywhere, much less was that information made public (something that Apple does, for example), nor on its engineering and technology blog; moreover, I had no idea how to even begin programming a color extraction algorithm that was at least similar. I had a purely intuitive idea of which color was chosen based on how the album covers looked, but that would hardly be enough.

I started looking at a few examples, hoping something would come to mind—a starting point.

Can I avoid fatigue?

I thought the images were probably processed server-side since the colors remain (for the most part) across different platforms, but I was also sure I hadn’t found anything about it in the documentation, so I went to the web player to scrutinize the requests and responses and, yes, indeed, I found that when the request is made to get a song’s lyrics, the response also includes the background color they should have—the color I was looking for.

The difference is that the endpoint the request goes to doesn’t have much to do with the official API; I imagine they’re internal services that only first party applications can use. The endpoint looks like https://spclient.wg.spotify.com/color-lyrics/v2/track/{track_id}/image/{image_url} and only returns the expected color if Spotify (or Musixmatch) has a record of the corresponding lyrics or if they are available for the region where the account is registered and if the image to be processed is also in their database; otherwise, there is neither lyric nor color.

I just had to figure out how to get the token required to automate the process, simulate being a web client from the terminal, and get all the colors I might want. The impediment was my laziness in finding patterns among the 10 headers the requests had and the death sentence threat Spotify gave me, so I didn’t continue down that path. It seemed promising.

Trying to understand Spotify

I had to devise some other way, so I tried to understand what color Spotify chooses for the backgrounds. Maybe the most colorful one? Maybe the one that occupies the largest region in the image? Maybe the one that stands out the most? Perhaps a combination of all the above and even more factors?

Let’s look at a few images along with their selected background color:

• Figure 1: seems easy; the most vibrant color (or the one farthest from black and white, as I understand it) is red and it’s not in a negligible proportion. Resulting color: red.
• Figure 2: let’s assume neither the white nor the gray from the top lyrics will be chosen, leaving magenta, yellow, light blue, and even green as options, but none stands out too much. Resulting color: pink.
• Figure 3: I expected it to be black, but apparently the algorithm avoids it at all costs, same as white. Resulting color: gray.
• Figure 4: here there are problems; I rule out black, but the red/orange stands out against the blue; it seems blue isn’t so discriminated against, but it leaves me doubtful. Resulting color: blue.
• Figure 5: well, the hypothesis of the most colorful color is holding up, although yellow might have been discarded for being dull or not representative enough. Resulting color: magenta.
• Figure 6: it gets difficult; there are at least three colors that could be considered vibrant in some way, but apparently, among them, the darkest one is chosen. Resulting color: blue.

If we also consider the first images in this article, the idea generally holds true except for two cases. In the second case, I would say the predominant color, in every aspect, should fluctuate between orange and red, but the officially chosen background is blue; much the same with the third case, the color that stands out is a sort of orange but also results in blue.

I haven’t been able to determine it completely yet, but I think it goes more or less like this:

• The image will be divided into N groups of similar colors to avoid dealing with potentially hundreds of slightly different colors. This helps the algorithm’s performance.
• The most colorful or vivid colors (defined later) will always be preferred over the rest. This preference factor will be defined by qC.
• All else being equal regarding how colorful a color is, the darkest color (i.e., the least luminous) wins. This preference coefficient will be stored in qD.
• The percentage of the image occupied by the color will be taken into account but won’t be given as much relevance as its level of vividness. This relevance will be defined by qR.
• If the resulting color is white or black, a lighter or darker gray will be chosen instead, depending on the color obtained previously.
• The final obtained color will have its saturation increased by a factor S to make it contrast more with the color white.

In pseudo-code, ignore the handwriting:

Trying to copy Spotify

Now, how to calculate the (excuse the word) “vividness”? With RGB coordinates it’s a bit complicated, so we’ll translate the colors to the CIELAB color space, which allows us to perform this type of mathematical operation more simply and without necessarily resorting to trigonometry. It’s like moving from Cartesian coordinates to polar coordinates. In this link you can find a guide from X-Rite and PANTONE to better understand how colors are expressed mathematically in their different systems.

According to theory, how vivid a human perceives a certain color depends on its saturation and its luminance, as follows:

\begin{aligned} V = (L^*)^2 + (C^*ab)^2 \end{aligned}

Where L* is luminance and C*ab is saturation

But I ignored it after discovering that focusing on saturation (C*ab) as the first component gave me better results.

However, for the luminance calculation we will make use of the RGB space, since it’s simpler that way. The equation (described in this manual from the UN telecommunications agency in section 3.2) is:

\begin{aligned} Y = 0.2126 * R_{lin} + 0.7152 * G_{lin} + 0.0722 * B_{lin} \end{aligned}

Where each RGB component is expressed linearly—or without gamma encoding—

I used the values 10 for the number of groups or clusters and 5.0, 1.5, and 0.8 for qC, qD, and qR, respectively, after doing some quick tests seeing which color the algorithm returned and which color Spotify put in its applications. It worked well empirically, but perhaps there were coefficients that worked better and would have taken me a long time of trial and error to find. Besides that, the formula I had come up with might or might not be a good alternative for the calculation I needed. It was time to use statistics to back up my assumptions.

Trying to improve the copy

With the help of the forbidden endpoint that Spotify didn’t want me to use, which I mentioned a few lines above (which I used with an alternate account to avoid dying), I put together a quick script to get key data from about 100 moderately random songs (though surely influenced by my musical tastes), which allowed me to subject the album covers to be processed by the algorithm and compare the result with the official color, apart from the three aspects that contribute to the final score without taking the qX coefficients into account.

Something more or less like this, although with many more rows, with 10/20 results per id (depending on N) and more decimals in the numbers:

chroma refers to colorfulness in the pseudocode. It seemed more appropriate.

In R I was able to perform a couple of linear regressions to find the best coefficients, at least for the 100 songs in question, giving more weight to cases where a higher score coincided with a lower color_delta, which means the formula works (or is on the right track) and removing relevance from the rest of the results for each group of colors belonging to an image.

But the formula might not be the most suitable for the calculation, so I also tried a slight change:

  S = qC * chroma + qD * darkness + qR * dominance =>  S = chroma ^ qC + darkness ^ qD + dominance ^ qR

Note the change from * to ^, moving from a linear to an exponential formula.

And I generated data again. The format was the same as the table above, but the data, of course, changed. I decided not to beat around the bush too much and start with values quite similar to those from the linear formula, 5.0, 1.2, and 0.9, and trust the math.

In this way, however, a linear regression would no longer be the appropriate analysis for the data, so after some research I went for the non-linear least squares method (or NLS for friends).

After a few rounds of fitting I put the results in a small table to better choose the path I would take.

It’s clear that neither of the two formulas, whether using 10 or 20 color groups, is very accurate. Most likely, I think the Spotify engineers take into account more variables than I do with this algorithm, although I can’t think which ones. It’s also possible that the formula itself is considerably more complicated, of course.

But it doesn’t matter; in that entire sea of mediocre numbers there is a method that is surely less bad than the rest, so I’ll go to war with it even if it only has a 31% success rate.

The resulting code for the adjusted linear formula with 20 clusters would end up being this:

import math
import numpy as np
from PIL import Image
from sklearn.cluster import KMeans

qC = 4.9226
qD = 1.4060
qR = 0.7932

def calculate_dark_colorfulness(rgb):
    red, green, blue = [x / 255.0 for x in rgb]

    red_greenness = red - green # or the a component
    yellow_blueness = (red + green)/2 - blue # or the b component. red + green output yellow in additive color (light)

    chroma = math.sqrt(red_greenness ** 2 + yellow_blueness ** 2)

    """
    Choose any of these three, bascially according to preference, the output is not going to change
    significantly. If you care about accuracy, you're free to do some research to determine the best
    option for your use case. Keep in mind that the overall formula coefficients might need to change too.

    They're as follows:

    First option: the standard way of obtaining luminance from RGB coordinates but without linearizing
    Second option: same as the previous, but linear (removing gamma correction)
    Third option: Independent HSP color space "percieved brightness" (https://alienryderflex.com/hsp.html)

    """

#    luminance = 0.2126 * red + 0.7152 * green + 0.0722 * blue # Relative luminance
#    luminance = 0.2126 * (red ** (1/2.2)) + 0.7152 * (green ** (1/2.2)) + 0.0722 * (blue ** (1/2.2))# Relative luminance without gamma correction
    luminance = math.sqrt(0.299 * (red ** 2) + 0.587 * (green ** 2) + 0.114 * (blue ** 2)) # Percieved brightness (HSP)
    darkness = 1 - luminance

    return (chroma, darkness)

### TO-DO: Implement improving picked color contrast with white, as lyrics are always white, to better match Spotify's choice
def improve_white_contrast(rgb, brightness_factor=0.85, saturation_boost=1.1):
    red, green, blue = [x / 255 for x in rgb]
    hue, sat, value = colorsys.rgb_to_hsv(red, green, blue)

    sat = min(sat * saturation_boost, 1.0)
    value = max(value * brightness_factor, 0.0)  # darken

    red, green, blue = colorsys.hsv_to_rgb(hue, sat, value)
    return tuple(int(x * 255) for x in (red, green, blue))

def extract_color_clusters(num_clusters):
    image = Image.open("./cover.jpg").convert("RGB")
    width, height = image.size

    try:
      pixels = np.array(image).reshape(-1, 3) # Turn a RGB matrix into an RGB 2D array

      # Cluster colors using K-Means
      kmeans = KMeans(n_clusters=num_clusters, random_state=0, n_init="auto")
      kmeans.fit(pixels)
      labels = kmeans.labels_

      colors = []

      # Group colors by cluster and calculate average for each
      clusters = [[] for _ in range(num_clusters)]
      for i, label in enumerate(labels):
        clusters[label].append(pixels[i])

      for group in clusters:
        color = np.mean(group, axis=0)
        chroma, darkness = calculate_dark_colorfulness(color)
        dominance = len(group)/(width * height)

        score = chroma * qC + darkness * qD + dominance * qR

        colors.append({
          'color': tuple(int(c) for c in color),
          'score': score,
        })

      return colors
    except Exception as e:
      print(e)
      print("Black and white image?")
      return [{
        'color': (128,128,128),
        'score': 10.0,
      }]

def get_best_color():
  return max(extract_color_clusters(20), key=lambda c: c['score'])

if __name__ == '__main__':
  print(get_best_color())

Code also available in this gist on GitHub.

Of course, the final formula becomes:

\begin{align} S = 4.923 * Ch + 1.406 * Dk + 0.793 * Do \end{align}