[REQ] Sonic Analysis

[BigInJapan]

About three years ago, Plex introduced a feature called Sonic Analysis as a paid feature available via their side project, Plexamp. It scans the music library and creates a sonic fingerprint of sorts for each track. That fingerprint is then used by various playlist building engines to create mixes of songs that sound similar.

The Sonic Adventure engine is particularly impressive. You give it two songs, and it maps out x tracks that will gradually walk you from the first song to the last sonically. So start it with a loud rocker, have it end it on a soft ballad, ask it for 10 tracks, and instead of a mix of random rockers and ballads, you get a mix where the first few are rockers, the next few are midtempo, and the last few are ballads, all sequenced out.

My library has over 15k tracks in it. I've been meticulously tagging it in MM for over a decade, and I've got countless AutoPlaylists in MM that mix those tracks in all sorts of interesting ways. But more and more, when I just want a good, quick mix, I find myself going to Plexamp as my player. The mixes that Sonic Analysis creates are just better than mine. They sequence songs in ways that all of my other tagging/searching/sorting just can't replicate.

The write-up is here: https://support.plex.tv/articles/sonic-analysis-music/

I know it's a stretch, but I would love to see the MM team take on building out a competing feature. Layering that sort of sonic scoring on top of all of the complex filtering we can already do in MM would be something really compelling.

[Peke]

Hi,
As you said, it is proprietary feature of PLEX and PlexAmp and MM can't use it https://support.plex.tv/articles/sonic-analysis-music/

From what I read https://joelezekielconfetti.wordpress.c ... omponents/ it is that PLEX used Sonic as a fancy name to already known https://mixedinkey.com/harmonic-mixing-guide/ MM already have support for Initial Key, BPM, even track Energy thru extended tags. Only thing missing a script developer that would be willing to incorporate it into MM and AutoDJ script. Even as paid feature like PLEX.

Only downside is that MM can't calculate those fields itself, but can read tags written by other apps.

[rusty]

Thanks for the feedback! This is something that we've discussed internally and there's certainly room to improve the Auto-DJ to allow it to generate playlists based on more than just existing Auto-playlists.

Our focus atm is on getting MM 2024.0 released, but something along these lines is on our radar.

-Rusty

[BigInJapan]

Thanks. I appreciate you taking the time to review this request.

I hadn't come across Mixed in Key before. Looks interesting. Their site is a little light on technical details. It seems like it writes its findings to ID3 tags, but I couldn't find details on exactly which tags it uses. Do you happen to know? (No worries if not, just curious.)

Even without a new Auto DJ script, I can imagine ways those tags would be useful in auto-playlisting.

[Peke]

Hi,
Mixed in Key, can be set to write to Initial Key Tag, that MM natively support and rest can be remapped to Custom or extended tags.
See my setting in Mixed in Key:

[BigInJapan]

Thanks. That looks really promising.

I'm down the rabbit hole on this a little bit now, and I came across this project: https://www.beatunes.com/en/index.html that also seems to do key detection. Mixed in Key seems DJ targeted, whereas beaTunes looks like it's doing it more in a media player context, and it costs a little less.

Both have free trials, so I might do some playing around.

[Peke]

Hi,
Have not tested beatunes and I am curious what Tag data it saves that can MM use.

[BigInJapan]

I made a little time to play with this today. I'm more intrigued by how to get my Key and BPM tags populated at the moment than I am on how to make a playlist from that, so I looked around at some more methods of getting key analysis. Most tools I saw also did bpm and mood. I got the sense several were pulling from some common back-end. I then found this - https://essentia.upf.edu/ - which has some familiar logos at the bottom.

As for beaTunes, it seems to be populating the standard tags for BPM, Initial Key, and Mood. After I analyze the files there, I see the values in these fields in MM. I also see A LOT of extended tags.

On the beaTunes front end, we see a lot more analysis columns, like Tuning, Time Signature, Measures per Minute, Secondary BPM, Color, Timbre, Instrumentation, and Danceability. Those seem to be derived from some of the extended tags, and they can be leveraged in various ways in the beaTunes playlist creation tools. I assume a MM plugin would be able to also leverage these extended tags, but obviously they don't help for a standard MM AutoPlaylist.

The beaTunes documentation says much of this is coming from database lookups, not actually being computed on my machine. It seems like maybe it can analyze BPM and Key locally when there's no match. It returns these for every track. I threw some concert bootlegs at it that would be unlikely to match anything in a database, and I still got values back for these. It seems to also always calculate Color, Timbre, and Danceability values. The other columns are sometimes blank, more often so on obscure albums and concert bootlegs.

I'm curious to compare the key and bpm values to what Mixed in Key returns. I'd like to do that on a spreadsheet. Do you happen to have a recommendation for a tool to bulk export tags to a CSV?

[BigInJapan]

Ha, never mind that last q. Spaced and forgot we have "File list" in MM.

One other observation - I see "essentia" in the extended tags from beaTunes, so it looks like it's using those tools too.

[BigInJapan]

I ran my set of test songs through Mixed in Key to compare. In a lot of cases, the two returned the same key and similar enough BPM scores. In the places where they differ, it's hard to really say which is more correct. One test I thought would shed light was cases where I have multiple recordings of the same song (studio release, concert bootlegs, official live releases, greatest hits release). But no particular trend emerged from that. Typically, if I had say four versions of a song, both tools would put 3 out of 4 in the same key, but they'd differ on which one they put in a different key.

There's a lot of nuance in both measures, and I had no expectation both tools would give exactly the same answer. They matched up often enough, and the differences seamed reasonable enough, that in terms of accuracy, I think I'd be comfortable using either. I think my biggest concern was that beaTunes, being a bit of an obscure project, might have just been faking its funk for attention. That definitely does not seem to be the case. The only failing I found in it, a few of my albums were encoded with Nero tags at some point, and beaTunes can't write those tags. I think I understand that Nero tags were a dead end anyway. I'll have to re-encode those albums to set their tags in beaTunes (thankfully a pretty painless operation in MM). Mixed in Key had no problem writing those (nor does MM, for that matter!).

I found three issues with Mixed in Key
1. It will not analyze a track that's >15 min. Not sure why it doesn't just use the first 10 minutes or whatever. It simply skips over these.
2. It writes a non-standard, proprietary value to the Initial Key tag.
3. At $60, it's a bit pricey if Key and BPM tagging are the only thing one is using it for. Perhaps reasonable if using the other features, but steep for my use case.

About that non-standard value... the ID3 spec says Initial Key should be populated with a string in standard music notation. in beaTunes, they store the key this way, and then give the option of having the UI translate that to Open Key at the display layer. Open Key is an open-source version of DJ friendly key notation. Mixed in Key uses a proprietary notation method called Camelot, and it writes the Camelot values directly to the tag rather than translating to that in the UI.

[Lowlander]

If you're trying to get BPM and Initial Key you should check out Mixxx as it can do both.

[BigInJapan]

Right on. I'll take a look. Guessing by the name this is also a DJ tool. Seems like this was a bigger priority in that world.

Do you happen to know if it writes any other tags? Or if its key detection is also Essentia based?

[Lowlander]

No clue. I've just used it for BPM and Initial Key.

[BigInJapan]

I gave Mixxx a shot. On the upside, it successfully tagged every song in my test set. And it's truly free free. It does not seem to provide any elements beyond Initial Key and BPM. In Initial Key, it looks like it's using special characters for flat and sharp, which looks nice, but isn't technically what's spec'd for that tag.

You opened my eyes that this is old news in DJ circles, so I went searching r/DJs. This post makes a pretty convincing case that Mixed In Key returns 15% more accurate key detection than Mixxx or beaTunes. That compared the results against a trained human ear. I don't have that to work with in my tests, so I've only compared against each other.
beaTunes matches Mixed In Key = 69%
Mixxx matches Mixed In Key = 67%
beaTunes matches Mixxx = 64%

All told, I've convinced myself it's worth the $60 to get the key detection from Mixed in Key. Before I saw the Reddit post, I was ready to tell myself Mixxx was close enough given the cost. But putting that specific "15% more accurate" factoid in my brain changed my mind.

As much for my own future recall as anything, here's a summary of the tools I looked at:

Mixed In Key
- Most expensive by far, but seems legitimately more accurate in key detection
- Refuses to tag songs longer than 15 min
- Can write the key tag using standard text or proprietary Camelot notation
- Provides an "Energy" score in an extended tag
- Focused on tagging songs for use in other applications. No other particularly useful features of its own.

beaTunes
- Mid priced, and key detection accuracy is probably good enough
- Stumbled with some one-off albums that were encoded with the defunct Nero encoder
- Can write the key tag using standard text or Open Key notation
- Provides LOTS of extended tags with other sonic attributes (color, instrumentation, secondary bpm, etc). Also populates Mood tag with text strings, but these often seem nonsense.
- Includes interesting playlist builder tools leveraging the extended tags it creates during analysis

Mixxx
- Fully free, and key detection accuracy is probably good enough
- Had no issues with any files in my test set
- Uses special characters for flats and sharps in key tag rather than standard text
- Does not provide any sonic attributes beyond key and bpm
- Lots of other DJ-centric functionality

So now that I'll reliably have Initial Key and BPM, I might be motivated to work on an Auto-DJ script that leverages them. I've never tried scripting MM, but I've worked in js enough to get by. You seem to have plenty of documentation to get me started. I peeked at the "Playcount > X" sample, and I get the gist. I guess I'll bring it up over on the developer forum if I get rolling on it.

Seems there are interesting opportunities for playlist builder add ins around these tags too. An Auto-DJ script seems like an easier place to start playing with the idea though.

Thanks for your input on this. It'd still be great if MM someday had its own tools for detecting sonic attributes, but I appreciate the pointers on bringing that in from other sources.

[Peke]

Hi,
Hmmm... Nice article. I tested with similar results.
I also added https://www.abyssmedia.com/bpmcounter/ to mix (even I do not own all apps listed in the redit)




Difference for my sample track is 0.09 BPM, I came to conclusion that depending In genre and track energy BPM differs from app to app due the base line calculation. Personally I use Mixed in Key for years and it is far from perfect, but BPM counter also works great too and is free like Mixxx. I personally always combine two/three apps to be sure.