Skip to main content

How big are changes in Discogs (2)?

Recently Discogs changed its data dump format, making it more difficult to compare releases the way I used to do and which is described in a previous blog post.

So, the only thing I could do: dive into the XML a bit more and compare elements in the XML output for releases that have been changed. The results are quite interesting.

I took the data dumps of March 1 2020 and April 1 2020, split these data dumps into individual XML files, computed SHA256 checksums for each of these files and ignored the ones that were the same in both dataset, leaving me with 531,444 releases to look at:

Releases that were changed in March 2020

This graph is the same graph that I have been seeing for the last 2.5 years, but what I wanted to know is: what are the releases with relevant changes and where are those changes? Personally I don't think that the YouTube videos are very relevant, and since that was the biggest change in the Discogs data dumps I decided to filter these and then compare.

What I did is the following:

  1. read the XML for the release in both data dumps
  2. compare each element that is a direct child of the top level "release" element
  3. ignore the "videos" element
  4. record if any elements were added, removed or changed

As it turns out quite a few releases that were marked as different actually are identical if you ignore the "videos" (I think the YouTube videos are irrelevant information): 224,352 releases were actually identical when ignoring the YouTube videos. That leaves 307,092 files that had relevant changes:


Releases that had relevant changes in March 2020

The graph is interesting to see, as it is quite different to the other graph that I am quite familiar with. It seems like most of the changes in old releases are actually only for YouTube videos:

Releases with only YouTube video changes in March 2020


What iI wanted to know next is where these changes were made in the releases. I recorded three types of changes:
  1. element removed
  2. element added
  3. element changed
and also which element was changed.

There were 29 different change/element pairs. From most to least changes:

  1. changed identifiers: 69427
  2. changed tracklist : 63399
  3. changed companies : 62219
  4. changed images : 59756
  5. changed extraartists : 54986
  6. changed data_quality : 42555
  7. changed formats : 40751
  8. changed notes : 33513
  9. changed labels : 31066
  10. added master_id : 27111
  11. added images : 15907
  12. changed artists : 15472
  13. changed title : 14463
  14. added notes : 12813
  15. changed styles : 10165
  16. changed released : 7535
  17. changed genres : 6597
  18. added released : 6214
  19. added styles : 5759
  20. changed master_id : 5139
  21. changed country : 4069
  22. removed notes : 3495
  23. removed released : 2353
  24. added country : 1629
  25. removed images : 1047
  26. removed master_id : 898
  27. removed styles : 329
  28. removed country : 260
  29. added genres : 1

So the most common type of change is definitely "identifiers", which translate to "Barcode and Other Identifiers" on the Discogs webpage. This is not entirely unexpected. What is interesting to see is that very few sections are completely removed and if a section is removed completely it is mostly "notes" (which makes sense, as there are plenty of releases with bogus notes) and "released" (released dates/years).

There are probably still some other tweaks possible here. For example, personally, I think that an element such as "data_quality" doesn't add much information, except which releases to possibly ignore (I have ranted about this before) and there were a bit over 12,500 releases where the only change was this particular field. Others that I would probably also ignore: releases where only "genres" (264 releases) or "styles" (1480 releases) were changed.

My next goal will be to dive a bit deeper into how big the differences are using TLSH. I will leave that for next time.
Labels:

Comments

Post a Comment

Popular posts from this blog

SID codes (part 1)

One thing that I only learned about after using Discogs is the so called Source Identification Code, or SID. These codes were introduced in 1994 to combat piracy and to find out on which machines a CD was made. It was introduced by Philips and adopted by IFPI, and specifications are publicly available which clearly describe the two available SID codes (mastering SID code and mould SID code). Since quite a few months Discogs has two fields available in the " Barcode and Other Identifiers " (BaOI) section: Mould SID code Mastering SID code A few questions immediately popped up in my mind: how many releases don't have a SID field defined when there should be (for example, the free text field indicates it is a SID field)? how many releases have a SID field with values that should not be in the SID field? how many release have a SID field, but a wrong year (as SID codes were only introduced in 1994) how many vinyl releases have a SID code defined (which is impossi
1 comment
Read more

SPARS codes (part 1)

Let's talk about SPARS codes used on CDs (or CD-like formats). You have most likely seen it used, but maybe don't know its name. The SPARS code is a three letter code indicating if recording, mixing and mastering were analogue or digital. For example they could look like the ones below. There is not a fixed format, so there are other variants as well. Personally I am not paying too much attention to these codes (I simply do not care), but in the classical music world if something was labeled as DDD (so everything digital) companies could ask premium prices. That makes it interesting information to mine and unlock, which is something that Discogs does not allow people to do when searching (yet!) even though it could be a helpful filter. I wanted to see if it can be used as an identifier to tell releases apart (are there similar releases where the only difference is the SPARS code?). SPARS code in Discogs Since a few months SPARS is a separate field in the Discogs
Post a Comment
Read more

Country statistics (part 2)

One thing I wondered about: for how many releases is the country field changed? I looked at the two most recent data dumps (covering February and March 2019) and see where they differed. In total 5274 releases "moved". The top 20 moves are: unknown -> US: 454 Germany -> Europe: 319 UK & Europe -> Europe: 217 unknown -> UK: 178 UK -> Europe: 149 Netherlands -> Europe: 147 unknown -> Europe: 139 unknown -> Germany: 120 UK -> US: 118 Europe -> Germany: 84 US -> UK: 79 USA & Canada -> US: 76 US -> Canada: 65 unknown -> France: 64 UK -> UK & Europe: 62 UK & Europe -> UK: 51 France -> Europe: 51 Saudi Arabia -> United Arab Emirates: 49 US -> Europe: 46 unknown -> Japan: 45 When you think about it these all make sense (there was a big consolidation in Europe in the 1980s and releases for multiple countries were made in a single pressing plant) but there are also a few weird changes:
Post a Comment
Read more