Skip to main content

How big are changes in Discogs (1)?

One thing I keep asking myself is how big changes in Discogs releases typically are: are the changes just small tweaks, or are they really big changes? I decided to compare the size of the changes of releases in two months using a very simple method: looking at the size of the differences.

I decided to not go for a very advanced approach that would require me to parse the XML and compare each field, but instead go fora much simpler approach and look at the size differences of the raw XML data.

I took two data dumps from the Discogs data website (namely discogs_20191003_releases.xml.gz and discogs_20191101_releases.xml.gz). I split these two dumps into individual XML files (one per release) and renamed them to have the right release number in the file name. I then compared the files with a regular cryptographic hash (SHA256) to see which files were different.

These files (439,651 in total) were compared using a locality sensitive hash, specifically TLSH as I am quite familiar with it. This hashing algorithm allows you to compare two hashes and compute a number. The lower the number that TLSH gives you, the closer the two pieces of data are. Translating it to the Discogs domain: the lower the TLSH number, the smaller the differences between two releases. So this allows you to compute a distance.

Based on previous experience with TLSH I would say that anything with a score of 20 or lower would mean that the two pieces of data are very close. My gut feeling was that a lot of the changes in Discogs would be actually very small and it turns out that this is indeed the case. The top 15 of the number of releases for each TLSH distance found gives me this:

  1. distance: 1, releases: 25858
  2. distance: 5, releases: 14788
  3. distance: 4, releases: 14611
  4. distance: 6, releases: 14150
  5. distance: 3, releases: 13893
  6. distance: 7, releases: 13608
  7. distance: 8, releases: 12611
  8. distance: 2, releases: 12288
  9. distance: 9, releases: 11291
  10. distance: 10, releases: 10189
  11. distance: 11, releases: 8993
  12. distance: 12, releases: 7791
  13. distance: 13, releases: 6933
  14. distance: 19, releases: 6848
  15. distance: 18, releases: 6813
The nr 1 spot is about 5.8%. The top 5 is 18.9%, the top 10 almost 32.6% and the top 15 a little bit over 41%. Adding the numbers for TLSH distances 14, 15, 16, 17 and 20 gets me to almost 47%.

So indeed: a very significant portion of the edits in Discogs are small edits. I haven't yet looked at where in the database these small edits happen and if older releases see bigger or smaller edits than newer releases. I also don't know yet which parts of a release are updated the most. I will leave that for another time.
Labels:

Comments

Post a Comment

Popular posts from this blog

SID codes (part 1)

One thing that I only learned about after using Discogs is the so called Source Identification Code, or SID. These codes were introduced in 1994 to combat piracy and to find out on which machines a CD was made. It was introduced by Philips and adopted by IFPI, and specifications are publicly available which clearly describe the two available SID codes (mastering SID code and mould SID code). Since quite a few months Discogs has two fields available in the " Barcode and Other Identifiers " (BaOI) section: Mould SID code Mastering SID code A few questions immediately popped up in my mind: how many releases don't have a SID field defined when there should be (for example, the free text field indicates it is a SID field)? how many releases have a SID field with values that should not be in the SID field? how many release have a SID field, but a wrong year (as SID codes were only introduced in 1994) how many vinyl releases have a SID code defined (which is impossi...
1 comment
Read more

SPARS codes (part 1)

Let's talk about SPARS codes used on CDs (or CD-like formats). You have most likely seen it used, but maybe don't know its name. The SPARS code is a three letter code indicating if recording, mixing and mastering were analogue or digital. For example they could look like the ones below. There is not a fixed format, so there are other variants as well. Personally I am not paying too much attention to these codes (I simply do not care), but in the classical music world if something was labeled as DDD (so everything digital) companies could ask premium prices. That makes it interesting information to mine and unlock, which is something that Discogs does not allow people to do when searching (yet!) even though it could be a helpful filter. I wanted to see if it can be used as an identifier to tell releases apart (are there similar releases where the only difference is the SPARS code?). SPARS code in Discogs Since a few months SPARS is a separate field in the Discogs ...
Post a Comment
Read more

Country statistics (part 2)

One thing I wondered about: for how many releases is the country field changed? I looked at the two most recent data dumps (covering February and March 2019) and see where they differed. In total 5274 releases "moved". The top 20 moves are: unknown -> US: 454 Germany -> Europe: 319 UK & Europe -> Europe: 217 unknown -> UK: 178 UK -> Europe: 149 Netherlands -> Europe: 147 unknown -> Europe: 139 unknown -> Germany: 120 UK -> US: 118 Europe -> Germany: 84 US -> UK: 79 USA & Canada -> US: 76 US -> Canada: 65 unknown -> France: 64 UK -> UK & Europe: 62 UK & Europe -> UK: 51 France -> Europe: 51 Saudi Arabia -> United Arab Emirates: 49 US -> Europe: 46 unknown -> Japan: 45 When you think about it these all make sense (there was a big consolidation in Europe in the 1980s and releases for multiple countries were made in a single pressing plant) but there are also a few weird changes:...
Post a Comment
Read more