That was where I built, for the first time, a graphical interface for my music project. This interface needed a colorful background for the image indicating which song was playing.
The problem is that, looking at the Spotify developer portal documentation, there was no reference to colors anywhere, much less was that information made public (something that Apple does, for example), nor on its engineering and technology blog; moreover, I had no idea how to even begin programming a color extraction algorithm that was at least similar. I had a purely intuitive idea of which color was chosen based on how the album covers looked, but that would hardly be enough.

I started looking at a few examples, hoping something would come to mind—a starting point.

Can I avoid fatigue?

I thought the images were probably processed server-side since the colors remain (for the most part) across different platforms, but I was also sure I hadn’t found anything about it in the documentation, so I went to the web player to scrutinize the requests and responses and, yes, indeed, I found that when the request is made to get a song’s lyrics, the response also includes the background color they should have—the color I was looking for.

The difference is that the endpoint the request goes to doesn’t have much to do with the official API; I imagine they’re internal services that only first party applications can use. The endpoint looks like https://spclient.wg.spotify.com/color-lyrics/v2/track/{track_id}/image/{image_url} and only returns the expected color if Spotify (or Musixmatch) has a record of the corresponding lyrics or if they are available for the region where the account is registered and if the image to be processed is also in their database; otherwise, there is neither lyric nor color.

I just had to figure out how to get the token required to automate the process, simulate being a web client from the terminal, and get all the colors I might want. The impediment was my laziness in finding patterns among the 10 headers the requests had and the death sentence threat Spotify gave me, so I didn’t continue down that path. It seemed promising.

Trying to understand Spotify

I had to devise some other way, so I tried to understand what color Spotify chooses for the backgrounds. Maybe the most colorful one? Maybe the one that occupies the largest region in the image? Maybe the one that stands out the most? Perhaps a combination of all the above and even more factors?

Let’s look at a few images along with their selected background color:

• Figure 1: seems easy; the most vibrant color (or the one farthest from black and white, as I understand it) is red and it’s not in a negligible proportion. Resulting color: red.
• Figure 2: let’s assume neither the white nor the gray from the top lyrics will be chosen, leaving magenta, yellow, light blue, and even green as options, but none stands out too much. Resulting color: pink.
• Figure 3: I expected it to be black, but apparently the algorithm avoids it at all costs, same as white. Resulting color: gray.
• Figure 4: here there are problems; I rule out black, but the red/orange stands out against the blue; it seems blue isn’t so discriminated against, but it leaves me doubtful. Resulting color: blue.
• Figure 5: well, the hypothesis of the most colorful color is holding up, although yellow might have been discarded for being dull or not representative enough. Resulting color: magenta.
• Figure 6: it gets difficult; there are at least three colors that could be considered vibrant in some way, but apparently, among them, the darkest one is chosen. Resulting color: blue.

If we also consider the first images in this article, the idea generally holds true except for two cases. In the second case, I would say the predominant color, in every aspect, should fluctuate between orange and red, but the officially chosen background is blue; much the same with the third case, the color that stands out is a sort of orange but also results in blue.

I haven’t been able to determine it completely yet, but I think it goes more or less like this:

• The image will be divided into N groups of similar colors to avoid dealing with potentially hundreds of slightly different colors. This helps the algorithm’s performance.
• The most colorful or vivid colors (defined later) will always be preferred over the rest. This preference factor will be defined by qC.
• All else being equal regarding how colorful a color is, the darkest color (i.e., the least luminous) wins. This preference coefficient will be stored in qD.
• The percentage of the image occupied by the color will be taken into account but won’t be given as much relevance as its level of vividness. This relevance will be defined by qR.
• If the resulting color is white or black, a lighter or darker gray will be chosen instead, depending on the color obtained previously.
• The final obtained color will have its saturation increased by a factor S to make it contrast more with the color white.

In pseudo-code, ignore the handwriting:

Trying to copy Spotify

Now, how to calculate the (excuse the word) “vividness”? With RGB coordinates it’s a bit complicated, so we’ll translate the colors to the CIELAB color space, which allows us to perform this type of mathematical operation more simply and without necessarily resorting to trigonometry. It’s like moving from Cartesian coordinates to polar coordinates. In this link you can find a guide from X-Rite and PANTONE to better understand how colors are expressed mathematically in their different systems.

According to theory, how vivid a human perceives a certain color depends on its saturation and its luminance, as follows:

\begin{aligned} V = (L^*)^2 + (C^*ab)^2 \end{aligned}

Where L* is luminance and C*ab is saturation

But I ignored it after discovering that focusing on saturation (C*ab) as the first component gave me better results.

However, for the luminance calculation we will make use of the RGB space, since it’s simpler that way. The equation (described in this manual from the UN telecommunications agency in section 3.2) is:

\begin{aligned} Y = 0.2126 * R_{lin} + 0.7152 * G_{lin} + 0.0722 * B_{lin} \end{aligned}

Where each RGB component is expressed linearly—or without gamma encoding—

I used the values 10 for the number of groups or clusters and 5.0, 1.5, and 0.8 for qC, qD, and qR, respectively, after doing some quick tests seeing which color the algorithm returned and which color Spotify put in its applications. It worked well empirically, but perhaps there were coefficients that worked better and would have taken me a long time of trial and error to find. Besides that, the formula I had come up with might or might not be a good alternative for the calculation I needed. It was time to use statistics to back up my assumptions.

Trying to improve the copy

With the help of the forbidden endpoint that Spotify didn’t want me to use, which I mentioned a few lines above (which I used with an alternate account to avoid dying), I put together a quick script to get key data from about 100 moderately random songs (though surely influenced by my musical tastes), which allowed me to subject the album covers to be processed by the algorithm and compare the result with the official color, apart from the three aspects that contribute to the final score without taking the qX coefficients into account.

Something more or less like this, although with many more rows, with 10/20 results per id (depending on N) and more decimals in the numbers:

chroma refers to colorfulness in the pseudocode. It seemed more appropriate.

In R I was able to perform a couple of linear regressions to find the best coefficients, at least for the 100 songs in question, giving more weight to cases where a higher score coincided with a lower color_delta, which means the formula works (or is on the right track) and removing relevance from the rest of the results for each group of colors belonging to an image.

But the formula might not be the most suitable for the calculation, so I also tried a slight change:

  S = qC * chroma + qD * darkness + qR * dominance =>  S = chroma ^ qC + darkness ^ qD + dominance ^ qR

Note the change from * to ^, moving from a linear to an exponential formula.

And I generated data again. The format was the same as the table above, but the data, of course, changed. I decided not to beat around the bush too much and start with values quite similar to those from the linear formula, 5.0, 1.2, and 0.9, and trust the math.

In this way, however, a linear regression would no longer be the appropriate analysis for the data, so after some research I went for the non-linear least squares method (or NLS for friends).

After a few rounds of fitting I put the results in a small table to better choose the path I would take.

It’s clear that neither of the two formulas, whether using 10 or 20 color groups, is very accurate. Most likely, I think the Spotify engineers take into account more variables than I do with this algorithm, although I can’t think which ones. It’s also possible that the formula itself is considerably more complicated, of course.

But it doesn’t matter; in that entire sea of mediocre numbers there is a method that is surely less bad than the rest, so I’ll go to war with it even if it only has a 31% success rate.

The resulting code for the adjusted linear formula with 20 clusters would end up being this:

import math
import numpy as np
from PIL import Image
from sklearn.cluster import KMeans

qC = 4.9226
qD = 1.4060
qR = 0.7932

def calculate_dark_colorfulness(rgb):
    red, green, blue = [x / 255.0 for x in rgb]

    red_greenness = red - green # or the a component
    yellow_blueness = (red + green)/2 - blue # or the b component. red + green output yellow in additive color (light)

    chroma = math.sqrt(red_greenness ** 2 + yellow_blueness ** 2)

    """
    Choose any of these three, bascially according to preference, the output is not going to change
    significantly. If you care about accuracy, you're free to do some research to determine the best
    option for your use case. Keep in mind that the overall formula coefficients might need to change too.

    They're as follows:

    First option: the standard way of obtaining luminance from RGB coordinates but without linearizing
    Second option: same as the previous, but linear (removing gamma correction)
    Third option: Independent HSP color space "percieved brightness" (https://alienryderflex.com/hsp.html)

    """

#    luminance = 0.2126 * red + 0.7152 * green + 0.0722 * blue # Relative luminance
#    luminance = 0.2126 * (red ** (1/2.2)) + 0.7152 * (green ** (1/2.2)) + 0.0722 * (blue ** (1/2.2))# Relative luminance without gamma correction
    luminance = math.sqrt(0.299 * (red ** 2) + 0.587 * (green ** 2) + 0.114 * (blue ** 2)) # Percieved brightness (HSP)
    darkness = 1 - luminance

    return (chroma, darkness)

### TO-DO: Implement improving picked color contrast with white, as lyrics are always white, to better match Spotify's choice
def improve_white_contrast(rgb, brightness_factor=0.85, saturation_boost=1.1):
    red, green, blue = [x / 255 for x in rgb]
    hue, sat, value = colorsys.rgb_to_hsv(red, green, blue)

    sat = min(sat * saturation_boost, 1.0)
    value = max(value * brightness_factor, 0.0)  # darken

    red, green, blue = colorsys.hsv_to_rgb(hue, sat, value)
    return tuple(int(x * 255) for x in (red, green, blue))

def extract_color_clusters(num_clusters):
    image = Image.open("./cover.jpg").convert("RGB")
    width, height = image.size

    try:
      pixels = np.array(image).reshape(-1, 3) # Turn a RGB matrix into an RGB 2D array

      # Cluster colors using K-Means
      kmeans = KMeans(n_clusters=num_clusters, random_state=0, n_init="auto")
      kmeans.fit(pixels)
      labels = kmeans.labels_

      colors = []

      # Group colors by cluster and calculate average for each
      clusters = [[] for _ in range(num_clusters)]
      for i, label in enumerate(labels):
        clusters[label].append(pixels[i])

      for group in clusters:
        color = np.mean(group, axis=0)
        chroma, darkness = calculate_dark_colorfulness(color)
        dominance = len(group)/(width * height)

        score = chroma * qC + darkness * qD + dominance * qR

        colors.append({
          'color': tuple(int(c) for c in color),
          'score': score,
        })

      return colors
    except Exception as e:
      print(e)
      print("Black and white image?")
      return [{
        'color': (128,128,128),
        'score': 10.0,
      }]

def get_best_color():
  return max(extract_color_clusters(20), key=lambda c: c['score'])

if __name__ == '__main__':
  print(get_best_color())

Code also available in this gist on GitHub.

Of course, the final formula becomes:

\begin{align} S = 4.923 * Ch + 1.406 * Dk + 0.793 * Do \end{align}

That’s where this little device comes in. It’s a Vobot Mini Dock, and I’ve been hacking on it to display Spotify information. I’ve been wanting to do this for a while, and it’s been a fun project.

It all started when I saw the Spotify Car Thing (RIP, that thing was cool) and thought, “Hey, I could do something like that, but smaller and for my desk!” It was supposed to be a fun little side project, something I could tinker with in my spare time. Little did I know…

Obviously, I couldn’t expect it to be as powerful or feature-rich as the Car Thing. It’s based on the ESP32S3, which is a tiny microcontroller with 512 KB of RAM and a 240 MHz CPU (for comparison, a Samsung Galaxy S25 has 12 GB of RAM and a processor that can reach 4.47 GHz). It also has a tiny LCD screen, 240x320 pixels. So, yeah, it’s very basic. But I figured, “Surely, displaying the time and a song title won’t require that much power, right?”

My initial goal was simple: get the Vobot to display the currently playing song, the artist, and, ideally, the album art. I searched the Vobot app store for something similar, but the selection was pretty limited, so I decided to roll my own.

The Problem(s)

The first hurdle was figuring out what programming language the Vobot used. I assumed it would be C++ or something similar. Nope! It uses MicroPython for the core logic and LVGL for the graphical user interface. I have some experience with Python, which was a good thing, but I’d never even heard of LVGL. So, I had to dive into the documentation and learn how to create buttons, labels, and images.

The LVGL documentation for C is pretty comprehensive, but the documentation for MicroPython felt a bit… lacking. I spent a lot of time just trying to get a basic “Hello, World!” app running. Plus, LVGL is a bit finicky.

I also had a few pre-existing projects that I could draw inspiration from. Thanks to @itandy848 and their HK Weather project, and the Vobot community itself, I managed to piece together a working solution.

One more thing: the official Vobot documentation for application development is pretty sparse. If they hadn’t open-sourced the code for some of their built-in apps, I probably wouldn’t have been able to figure out how to get started.

Alright, with all the pieces (or at least most of them) in place, I finally started writing the code.

Here’s the basic idea:

• Authentication: The Vobot needs to authenticate with Spotify to access user data.
• Song Information: I need to retrieve the currently playing song’s title, artist, and album art from Spotify’s API.
• Display: I need to display that information on the Vobot’s tiny screen.
• Real-time Updates: The clock needs to update regularly to keep the information current.

Things didn’t go exactly as planned. It turns out there were a few unexpected challenges.

The Images

As you can see, the first version had a problem with the album art. It was… cropped. It took me a while to figure out why. LVGL, especially on resource-constrained microcontrollers like the ESP32, is really sensitive to image processing. Resizing images on the fly is expensive.

So, I thought, “Maybe I can just ask Spotify for a smaller image!” Unfortunately, Spotify provides three image sizes: 64x64, 300x300, and 640x640. None of those were ideal for the Vobot’s screen. I ended up having to experiment with different resizing algorithms to find one that looked reasonably good without completely bogging down the device.

The Requests

Each time the clock needed to update (roughly every 3 seconds), it had to make a request to Spotify’s servers. Those requests took a few milliseconds, which isn’t terrible, but it added up. Also, the responses from Spotify were surprisingly large—often tens or even hundreds of kilobytes. That’s a lot of data for a tiny device with limited memory.

Another thing: I wanted the background of the screen to be the dominant color from the album art, just like Spotify does in its apps. That seemed like a nice touch. But even that required some extra processing power.

It quickly became clear that trying to cram all that logic onto the Vobot was going to result in a pretty mediocre user experience. The clock would be slow, the memory would fill up quickly, and the update lag would be noticeable. Not to mention my code was far from optimized.

Client and Server

So, I decided to offload some of the work to a more powerful machine. I set up a Python server on a virtual machine running on my main computer. This server handles image resizing, makes the requests to Spotify, and prepares the data for the Vobot.

The advantages were clear:

• Faster Processing: Image resizing and API requests are much faster on a server.
• Reduced Memory Usage: The Vobot only receives the data it absolutely needs.
• More Reliable: The server can handle errors and retries more gracefully.
• Easier Development: Python is a more flexible language for this kind of task.

Finally, I got everything working! The “play/pause” indicator shows whether something is currently playing, the progress bar moves smoothly, the song title and artist labels scroll if they’re too long, and the background color dynamically changes to match the album art.

As for how I managed to get the background color to match the album art? Well, I wrote a whole separate post about that. Check it out here.

The source code for both the Vobot application and the server is available on this GitHub gist.

A couple of days ago, a melody from a song I’d heard ages ago popped back into my head. I couldn’t recall anything about it except that somewhere in the song, the singer (or singers) mentioned the word “dangerous.”

I did what any normal person would do, I cleared my throat and started trying to imitate a piano and a synthesizer to the best of my poor abilities, so that Shazam might be able to help. I put a lot of effort into it, but I couldn’t get it to recognize the sound; it was probably because my attempts at imitating instruments are pretty awful.

To add a bit of humor and context, this is me trying to make myself sound as close as possible to the song:

After no luck with Shazam, I figured I’d try ChatGPT. Here’s what we managed to piece together, based on what I could remember:

• Dreamy male vocals
• A romantic, almost retro-futuristic vibe
• A synth melody that kind of carries the whole thing
• Simple piano chords
• A medium-speed tempo (~100 BPM)
• That feeling of electronic/synth-pop music
• A smooth, understated rhythm
• A cosmic and slightly melancholic atmosphere

But still, nothing. After all that, I hadn’t found the song. Some of the suggestions were interesting, and some even had a similar feel, but none of them were it. So I mentioned the problem to a couple of people, hoping someone might recognize it, but they just shrugged and said they couldn’t quite figure it out even when the sounds sounded (heh) familiar to them. Bad, bad luck.

I recalled the song had a simple structure, and I thought I had a pretty good idea of what it sounded like, so I decided to give GarageBand a shot , to see if I could make a melody as close as possible to the original to then feed it back to Shazam. I was (and still am) basically a GarageBand newbie at the time.

The project ended up looking like this:

I figured that maybe, just maybe, a few seconds of the track would be enough for it to recognize it. It was a long shot, but hey, you’ve got to try. And the track itself wasn’t super complex – just a piano, a synth, and a basic drum beat. I wasn’t even sure if the drum beat should’ve been played along with the piano, so I experimented with different options.

I ended up with something like this:

A second attempt with Shazam, and boom, there it was! I finally found what I was looking for. The song is called I Like Chopin by Gazebo, and it’s not exactly synth-pop, though it has some of those elements. It doesn’t have two male voices, but it does have a very distinctive sound, and it definitely doesn’t have the word “dangerous” anywhere in the lyrics (which is what I’d been searching for! Silly me). That’s probably why ChatGPT couldn’t help me. 😔

Here’s a sample of the song:

To wrap things up, just in case

I was curious whether it was really my fault that I couldn’t find the song with ChatGPT’s help, so I started a new session and gradually fed it all the corrected information. I even told it to focus on searching for songs that included the words “rainy days”, which are actually part of the lyrics, and even then it didn’t give me a satisfactory answer. It seems like it can’t “read” the lyrics of songs to accurately identify them.

Here’s how I tried to give ChatGPT better clues about the lyrics:

Here, I tried to help it out by specifying the genre:

And then, after I told it the title of the song, this is what it said:

It seems like it’s not the best resource for finding music, even though it’s helped me out a few times in the past.

The GarageBand project can be downloaded from this link.

Everything was going relatively well until it started going relatively poorly: I noticed that sometimes the movies or videos I store there had difficulty starting playback and/or continuing playback when I made jumps on the timeline. It wasn’t a huge problem, as it’s not common to make time jumps when watching a movie, but the issue had already caught my attention.

I first thought my hard drives were failing since this problem didn’t occur a couple of years ago when I moved my media library to them. So, I ran a couple of SMART tests, and everything seemed fine with them. Ruled out. I thought it might be a network bottleneck, but it was very unlikely since I observed the problem during local playback of the files. Additionally, I had recently upgraded my local network bandwidth to a respectable 10 Gigabit (more on this in another post in the future). Ruled out. It crossed my mind that one of the nodes (computers/servers) might be having issues when jumping from one part of the movie to another, but I didn’t see any excessive CPU usage in any case. Ruled out.

I ruled out a couple more options over days and weeks, but I noticed something interesting while deliberately trying to reproduce the error for diagnostic purposes. It didn’t matter how many minutes the jump was; whether I skipped 5 or 50 minutes, there was still a noticeable delay. More interestingly, I realized that if I didn’t hover the cursor over the timeline and jumped directly by placing it right at the point I wanted to skip to, the delay practically disappeared. Very interesting.

The problem

Reviewing the logs in the browser’s DevTools window, I noticed many requests made to the same endpoint; all started with “Thumbnail,” and then everything made sense: while scrubbing, i.e., moving the cursor over the timeline to see a small preview of what happens along it, dozens of requests were made that allowed Emby to obtain thumbnails for the preview. Thumbnails are lightweight, yes, but in the order of dozens, they significantly delay subsequent actions.

I couldn’t determine how many requests are made per pixel, perhaps per em, but they result in many in a short time while scrubbing and can become frustrating because the video resumes playback only when all of them (or at least a good percentage) have been completed, successfully or not.

The solution

Visiting the Emby forums, I saw that they recommended enabling the option to save thumbnails on the same disk where the content resides. It’s an option I had deliberately left disabled because I didn’t want to use space unnecessarily. Additionally, it was also recommended that each video file reside within its own directory and not have them all at the same level, as this way both the operating systems involved (Debian for the NAS and Windows for the client I was using at the time) and Emby itself would have less work finding and accessing files.

Let’s get hands on, then.

This initiated a process that, using FFmpeg, generates BIF files (created by Roku) that encapsulate snapshots taken at regular intervals throughout the video. So far, so good, but the process takes approximately 9 seconds per gigabyte on my hardware, and with several terabytes in tow, it was going to take a good few hours.

Some time later, with the process finished, I had double the files in my main movies/videos directory; each MP4 or MKV file had an analogous BIF file with the same name. If I recall correctly, they recommended not having so many files at the same level, and what I did doubled the original number—totally wrong. The good thing is that the next step would surely improve things.

I had to move each pair of files to their own folder; the bad thing is that Emby doesn’t get happy at all when files are moved. It doesn’t matter if you just put them inside another folder; it represents a change in their location, and that’s enough. The consequence of this is that Emby wouldn’t recognize the “new” files as files it already knew about, and references to the “old” ones would be invalidated, thus ruining playlists and favorites, for example.

Reading around, they recommended using NFO files because they could preserve important information about the files, and as long as they’re alongside the videos, that information will be available.

I did the test: enabled the option to save everything on the media disk in NFO format and moved a couple of movies to their respective new folders. I rescanned the folder in Emby, refreshed the metadata, but nothing—the movies in question had disappeared from the playlists they were in, as well as from favorites. The curious thing is that if I returned the files to their original location, everything went back to normal. Apparently, the correct way to remove items from these lists is by deliberately removing them through the applications or the API.

Clearly, I needed another way to preserve my favorites and lists. I also noticed that Emby seems to assign a unique identifier to each video but based on an index that increments as new files are added, which means that, by Emby considering the moved files as new, they also didn’t retain their IDs and obtained a new one.

The best I could think of was to resolve it through the names of the movies/videos. I was almost 100% sure that none would be repeated and that they would basically function as a unique identifier, but Emby doesn’t think the same way, so I would have to use its API to search by name and thus be able to reach the real ID (the new one) to be able to add the files back to favorites.

Something to keep in mind is that when searching through the API or GUI, Emby doesn’t return a single result even if there’s an exact match, so we have to refine the search. A simple string comparison should be sufficient.

import requests
import json

EMBY_IP = ""
API_KEY = ""
USER_ID = "" # In the format XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
PARENT_ID = "" # In case the videos need to be filtered by library

HEADERS = {"X-Emby-Token": API_KEY}

def get_favorites():
    url = f"{EMBY_IP}/Users/{USER_ID}/Items?Recursive=true&IsFavorite=true&ParentId={PARENT_ID}&IncludeItemTypes=Movie"
    response = requests.get(url, headers=HEADERS)

    if response.status_code == 200:
        items = response.json().get("Items", [])
        return [item["Name"] for item in items]
    else:
        print("Error fetching favorites:", response.text)
        return {}

def search_new_item_id(name):
    url = f"{EMBY_IP}/emby/Items?Recursive=true&SearchTerm={name}"
    response = requests.get(url, headers=HEADERS)

    if response.status_code == 200:
        items = response.json().get("Items", [])

        matching_items = [item for item in items if item["Name"].strip().lower() == name.strip().lower()]

        if matching_items:
            return matching_items[0]["Id"]
    return None

def re_add_favorites(old_favorites):
    for name in old_favorites:
        new_id = search_new_item_id(name)
        if new_id:
            url = f"{EMBY_IP}/Users/{USER_ID}/FavoriteItems/{new_id}"
            response = requests.post(url, headers=HEADERS)

            if response.status_code == 200:
                print(f"Re-added favorite: {name}")
            else:
                print(f"Failed to re-add {name}: {response.text}")
        else:
            print(f"Could not find new ID for: {name}")

if __name__ == "__main__":
    print("Fetching current favorites...")
    old_favorites = get_favorites()

    input("Move your files now and press Enter when done...")

    print("Re-adding favorites with new paths...")
    re_add_favorites(old_favorites)

But now the files come in trios, since we have the media file, the thumbnail file (BIF), and the metadata file (NFO), and moving them manually into their respective new folders can be tedious. It’s tedious.

To solve this, I used FileBot, which is apparently quite popular among users on the Emby and Plex forums for organizing files—moving, copying, deleting, and more. The downside is that it’s paid software, and worse, it’s a subscription. But considering that it’s not very expensive (priced at $6 per year at the time of writing this), that I’ll probably need to perform these kinds of operations regularly in the future, and that I was too lazy to write another script, I decided to give this community-proven tool a shot.

Using the “File to Folder Name” preset, we’re able to create a shared folder for the three files and make that their new home. In the example, there are no other types of files, but as long as they all have exactly the same name (excluding the extension, of course), they’ll be placed into the same directory.

Final thoughts

With the above changes, the performance improvement is noticeable in some areas and significant in others. For instance, when scanning video libraries, it seems to run a bit faster; when it comes to scrubbing, thumbnail previews on the timeline now appear almost instantly, and there’s hardly any wait time when jumping to a different point, no matter how it’s done. The results seem quite satisfying.

For some day… hopefully

• Write a script to perform the same process but for playlists. It shouldn’t be complicated; it’s quite similar to the code we already have and could be adapted with little trouble.
• While I can use FileBot on as many machines as I want, regardless of the platform (it supports Docker, Ubuntu, macOS, Windows, and even CentOS), it would be wise to eventually replace it with a basic script. For now, I’ve already paid for it and will use the program for these tasks, but for what I need just a few lines of code should be enough.

While on a trip, I rented a car and decided to record my way to some second-hand stores located on the outskirts of the city I was staying in, a good 90 minute road trip one-way and around the same for the return. When the moment arrived to transfer all the media to a safer storage medium, I got reminded every clip the cameras recorded was about 4GB in size for every 7 minutes. Didn’t care, it looked good.

Some weeks later, when wanting to transfer them to their final destination for multi-device offline viewing thanks to Immich, my home NAS, not only did they take quite some time to transfer, but I also noticed a couple of issues:

• As alluded to above, the cameras cut the recording when a single clip reaches about 4GB in size, DJI says they use FAT32 (which precisely presents this limitation), which explains this
• Given the settings I picked when recording (1080p 60 fps HQ), videos had a bitrate of around 80Mbps, or 80000Kbps if you will, which is more than double that of many Blu-ray conventional recordings and a little less that what could be found in 4K Blu-ray discs
• These two cameras have no built-in GPS, which is definitely not a problem for the majority of people and cases but given the nature of what I wanted to capture this time, I think it would’ve been a really nice to have

Let’s see…

FAT32

The first issue is not really that problematic, it may very well not even be an issue and instead more of a feature of this kind of device. The fact that DJI uses FAT32 and video files are capped at 4GB could come in handy when working with them and could even prevent corruption problems.

Transferring MPEG-4 files that are too big could be cumbersome. At more than 36GB per hour of video, that kind of transfer could take hours to complete between wireless devices and, it not being a video container format that plays along particularly well with errors and corruption makes it so that the process would have to be restarted when that happens, which is frankly tedious.

On the other hand, if errors occur during the recording or while storing the videos in the MicroSD cards, the fact that they are cut into smaller pieces favors the odds of recovery.

In a nutshell, may be inconvenient in a couple of aspects but no biggie. Next.

Bitrate

This is indeed a problem. Okay, not a problem per se but hear me out: I know the bigger the file the more information it holds inside and the better when the time to edit comes, no doubt, but I think in this case the size is not offering that much value.
With its 1/2.3-inch sensor, it’s not capturing that much light (comparatively, of course) at all and the added file size is translated to, I don’t know, noise? Still, having more information doesn’t hurt (or shouldn’t), but in this case I think I can get rid of a lot of bytes without sacrificing much image quality. At least that’s the reasoning.

With that in mind, a bitrate of 80000kbps is just too much for casual videos that we might want to share on social media or play on a phone from time to time to share with friends. It would be different had the goal been to play them often in a big TV, in that case I would’ve gone straight for 4K but I don’t have 20 MicroSD cards laying around to be able to keep and keep recording and not worrying.

So, given the problems their size presents to edit them, move them around or even play them, here’s where FFmpeg enters the game.

FFmpeg (pronounced /‘ɛfɛfɛm’pɛɡ/ 😉), helped me reducing the file sizes without a huge impact on image quality. These are the used parameters:

ffmpeg -hide_banner -loglevel error -hwaccel qsv -c:v h264_qsv -i {input} -c:v h264_qsv -global_quality 25 {output}

• -hide_banner prevents the banner, copyright related stuff and compilation options from being output as they’re generally not useful data
• -loglevel error shows errors
• -hwaccel qsv makes use of Intel Quick Sync Video hardware encoders which greatly accelerate the process
• -c:v h264_qsv specifies the codec, h.264 for compatibility reasons
• -global_quality 25 sets the quality level of the output file. The lower the number, the better the quality, so the idea is to have some sort of balance. I’ll expand on this later

Results are very promising, files are reduced to about 850MB from 4GB, savings of up to 80%.

Images are compressed to enhance the site’s performance. Original, full quality frames can be found here and here respectively.

But don’t blindly (heh) trust my or your eyes, there are image quality perception and comparison algorithms such as VMAF, developed by Netflix, or SSIM, developed by US universities, that can help us decide what are the best FFmpeg encoding parameters given our needs.

I chose SSIM because in order to use Netflix’ method (by the way, don’t let the opportunity of reading their blog pass by) I would’ve needed to complete a couple of prior steps, such as making sure the videos use the YUV color space, for example, which means encoding them first if that’s not the case. The laziness in me wins this time.

SSIM, the structural similarity index, works by simulating (or calculating) the perceived quality someone would have about an image or a set of them, including aspects like luminance and contrast. More than one way of calculating it exist, but I opted for FFmpeg because it comes built-in witht the binary.

ffmpeg.exe -i {input1} -i {input2} -lavfi ssim=stats_file={output} -f null -

This command produces a pretty long file where we can see the comparison between each frame of both videos, giving us a general value and partial values for each of the three components of YUV (maybe there wasn’t a need to convert the videos before using VMAF after all 😛).

Its structure is similar to this:

Doing a light statistic analysis of the results for both 24 and 25 as -global_quality parameters when issuing the FFmpeg command to reduce quality, 24 has a slight advantage, a margin that seems acceptable to me in favor of 25, which made me double down on the decision to pick 25 as the final number.

Next, some numbers courtesy of CalculatorSoup (looks like the free version of Wolfram Alpha couldn’t handle it) about how similar the processed videos are to the original one in average as well as the standard deviation in both cases. The higher the average and the lower the standard deviation (σ), the better.

Keeping in mind that, in general, an MSSIM value (mean SSIM) greater than or eaqual to 0.99 means two images are indistinguishable (Sec. 4.1 Luminance), the results appear to be satisfactory.

On the one hand, with quality 24, the higher of the two, we have a mean of 0.9862 with an σ of 0.0084. This means that, following the formula above, ~95% of the time both videos have a similarity level between 96.94% and 100% and that more than 99% of the time that level is between 96.1% and 100%. Furthermore, they are in average 98.62% similar. Pretty nice.

On the other hand, with quality 25, the mean is 0.9844 with an σ of 0.0109, which means that ~95% of the time both videos have a similarity level between 96.26% and 100% and that more than 99% of the time that level is between 95.17% and 100%. Besides, they are in average 98.44% similar. Not too shabby.

That difference sounds like a fair price to pay for saving some bytes here and there.

Was all this numeric fest unnecessary? Most likely, but I really hoped I can get some numbers supporting or disproving my decision; happily, something closer to the former happened.

GPS

I realized the lack of geolocation info on the videos metadata when I added them to Immich, where a little map indicates where they were taken if said info is available. Good thing that it offers a built-in metadata editor, albeit basic, which I used to save the GPS coordinates of the point where each recording started; I had to watch them all and accurately locate them with the help of Google Maps.

Once geolocated, Immich created an external Exif file for each video, all of them with the same name but different extension: xmp.

So far so good, despite the Exif metadata being external in this case, having the same file name helps with some video players recognizing them automatically so it shouldn’t be a problem; however, I wanted all the data to be embedded in the videos themselves, that’s better and easier to manage.

So I used ExifTool to solve that:

exiftool -gpsposition -c "%.7f" {input}

• -gpsposition -c "%.7f" extracts the GPS data in the “%.7f” format, which basically means an integer follow by 7 decimals (e.g. -31.8267316)

exiftool -TagsFromFile {input} -All:All -XMP -ExtractEmbedded -overwrite_original {output}

• -TagsFromFile takes the Exif tags stored in the files previously created by Immich
• -All:All takes into consideration all the tags it possibly can
• -XMP same as above but with XMP tags, which tend to be non-standard metadata defined by the camera or editing software vendor

There was a tiny little additional problem, unfortunately: apparently, DJI cameras need to be connected to a smartphone or tablet via Bluetooth after having been disconnected for a long period of time in order to get the right date and time. This caused the timestamps in my videos to be off by around 90 minutes. I don’t know the exact amount of time because I didn’t have a solid reference besides a little video I took with my phone, so I estimate that they still might be off by about 3 minutes tops give or take. Sounds okay.

Naturally, I had to change the file timestamps but also the ones found in the Exif tags, given that I only applied the change to the files some video players would still be able to see the error because they sometimes prefer the metadata and not the what the filesystem says. Having everything correctly time-coded also helps with organization since they use time to build timelines.

Changing the files timestamps was easy. In Windows (PowerShell) we could do something like:

(Get-Item {input}).LastWriteTime = {new_date}

Don’t forget that new_date must have the correct format, in this case I opted for MM/dd/yyyy HH:mm because that worked on my Windows install.

(we’re close, promise)

Buuuut, and this is a big but, the intention was to subtract 90 minutes from the creation/modification datetime present on the files, so I needed to extract first. Good thing is that it’s not a destructive operation, so I left it for later once FFmpeg and ExifTool had done their job.

One little thing before having the complete script: when it comes to exif tags, at lest here in this case, the timezone was always set to UTC and the format yyyy-MM-dd HH:mm:ss, so that had to be taken into account, although not for changing the timestamps Windows sees.

Well, with all that in mind and a couple of extra details, the script would end up like this (now with a bash version for completeness):

$originals_path = {originals_path}
$processed_path = {processed_path}

Get-ChildItem $originals_path -Filter *.mp4 | Sort-Object -Property Name | Foreach-Object{
    ""
    "Original file: $($_.Fullname)"
    "Reducing video bitrate"
    ""
    ""

    ffmpeg -hide_banner -loglevel error -hwaccel qsv -c:v h264_qsv -i $_.Fullname -c:v h264_qsv -global_quality 25 {original_file}

    # qs stands for Quick Sync
    $qs_file = (Get-Item "$processed_path\$($_.Basename).mp4")

    "Bitrate-reduced file: $($qs_file.Fullname)"
    ""
    
    # Get the current creation datetime (wrong creation date)
    $wrongCreationDateTime = "{0:yyyy-MM-dd HH:mm}" -f $($_.CreationTime)

    # Get the right creation datetime but in Windows format
    $rightDateTimeWindowsFormatted = [datetime]::parseexact($wrongCreationDateTime, 'yyyy-MM-dd HH:mm', $null).AddMinutes(-90).toString('MM/dd/yyyy HH:mm')

    # Get the right creation datetime but in Exif format (UTC with seconds)
    $rightDateTimeExifUTCFormattedWithSeconds = [datetime]::parseexact($rightDateTimeWindowsFormatted, 'MM/dd/yyyy HH:mm', $null).AddMinutes(300).toString('yyyy:MM:dd HH:mm:00')

    ""
    "Original file: $($_.Fullname)"
    "QS file: $($qs_file.Fullname)"
    "Wrong creation datetime: $wrongCreationDateTime"
    "Right creation datetime in Windows format: $rightDateTimeWindowsFormatted"
    "Right datetime in ExifTool UTC format with seconds: $rightDateTimeExifUTCFormattedWithSeconds"
    ""

    "ExifTool copying all tags from original..."

    exiftool -TagsFromFile $_.Fullname -All:All -XMP -ExtractEmbedded -overwrite_original $qs_file.Fullname

    # Extracting coordinates info from XMP files
    $coords = exiftool -gpsposition -c "%.7f" "$($qs_file.Fullname).xmp"
    
    ""
    $coords
    ""

    "ExifTool setting geoposition and updated timestaps..."

    exiftool "-CreateDate=$rightDateTimeExifUTCFormattedWithSeconds" "-ModifyDate=$rightDateTimeExifUTCFormattedWithSeconds" "-Track*Date=$rightDateTimeExifUTCFormattedWithSeconds" "-Media*Date=$rightDateTimeExifUTCFormattedWithSeconds" -gpsposition="$coords" -overwrite_original $qs_file.Fullname

    (Get-Item $qs_file).LastWriteTime = $rightDateTimeWindowsFormatted
    ""
    ""
    "================================================================================================="
}

#!/bin/bash

$originals_path = {originals_path}
$processed_path = {processed_path}

for filename in `$originals_path/*.mp4`; do
  echo
  echo "Original file: $filename"
  echo "Reducing video bitrate"
  echo
  echo

  $qs_filename =  $"$processed_path/$(basename "$filename")"

  ffmpeg -hide_banner -loglevel error -hwaccel qsv -c:v h264_qsv -i $filename -c:v h264_qsv -global_quality 25 $qs_filename

  echo "Bitrate-reduced file: $qs_filename"
  echo

  $wrong_datetime = $(date -r "$filename" +"%F %H:%M")
  $right_datetime = $(date -d ""$wrong_datetime"  90 minutes" +"%F %H:%M")
  $right_datetime_exif_utc_w_seconds = $(date -d ""$right_datetime"  300 minutes" +"%F %H:%M:%S")

  echo
  echo "Original file: $filename"
  echo "QS file: $qs_filename"
  echo "Wrong creation datetime: $wrong_datetime"
  echo "Right datetime: $right_datetime"
  echo "Right datetime in ExifTool UTC format with seconds: $right_datetime_exif_utc_w_seconds"
  echo

  echo "ExifTool copying all tags from original..."

  exiftool -TagsFromFile $filename -All:All -XMP -ExtractEmbedded -overwrite_original $qs_filename

  $coords = exiftool -gpsposition -c "%.7f" "$qs_filename.xmp"

  echo
  echo $coords
  echo

  echo "ExifTool setting geoposition and updated timestaps..."

  exiftool "-CreateDate=$right_datetime_exif_utc_w_seconds" \
          "-ModifyDate=$right_datetime_exif_utc_w_seconds" \
          "-Track*Date=$right_datetime_exif_utc_w_seconds" \
          "-Media*Date=$right_datetime_exif_utc_w_seconds" \
          -gpsposition="$coords" -overwrite_original $qs_filename

  touch -d "$right_datetime" $qs_filename

  echo
  echo
  echo "================================================================================================="
done

Final thoughts

All of this left me with much lighter videos, properly tagged regarding where they were captured and with files with the right timestamps. We went from an outlook described by the cover image to this:

This way is easier to edit, transmit or even just play them.

I know the more information there is in a video, the more flexibility it will offer when being processed; after all, computers still can’t programagically invent information out of nothing, right? That’s why I may end up using the actual original clips to merge them and obtain a larger video on which only then apply all the processes learned here so that a reasonably-sized video is generated.

What’s important is that if we gained knowledge we didn’t waste time, although the time invested in manually locating every piece of video will hardly be recovered, unless I can somehow extrapolate the GPS data and generate a data-based route which would later let me create a real-time map animation to overlay on top of the video or something like that.

Who knows. 😉

Why not use Premiere, Resolve or even Movie Maker?

Well, I think using FFmpeg is not only more versatile but more fun when dealing with multiple files and when not knowing exactly what to do, and I’m all about having fun while solving problems up to a certain point.
Of course, it’s also more difficult than using Resolve, which I typically use, but I sometimes feel more at home in the terminal and I was going to use it anyways because of ExifTool.
Besides, we use FFmpeg in this house, it’s free and open source and lets you fully use your GPU for encoding if you want. Besides, it has a pretty big community.

I’ll use Resolve for color correction, I don’t think I want to deal with FFmpeg when using LUTs. 😔

For some day… hopefully

• Improve the code, it can surely be improved, especially if we can prevent ExifTool from being called twice, given it’s a destructive operation and it consumes quite some time. Also can be a bit more elegant
• Color correct the video
• Remember to connect the cameras to my phone the next time I want to record something