Skip to main content

What happened in Discogs in December 2017?

A bit later than usual (I was swamped with work), but there is a new datadump, so it is time to dive into statistics again. It might be good to read one of the previous posts before reading this one.

Release statistics

I downloaded the latest database dump (the "January dump") containing data from December 1 to December 31 (inclusive). The previous dump had 9,217,123 releases, the new dump has 9,324,867 releases. That means 107,744 releases more in the database, which seems to be quite consistent with the previous months.
  • 3,916,654 releases stayed the same
  • 5,298,409 releases were changed
  • 109,804 were added
  • 2,060 releases were removed from the database
  • 128 releases had status Draft, Deleted or Rejected
  • 11 releases that were not Accepted were in both the December and January dumps
  • 2 releases were moved from Draft to Accepted
So what is immediately obvious is that like with the December dump there is again a very large amount of changed releases and like last time it is because of a change in the XML format. It looks like the previous change didn't work out well and they changed it again, now from:

<master_id is_main_release="false">Needs Vote</master_id>

to:

<master_id is_main_release="false">29615</master_id>

because apparently the wrong data was used for the XML dump (the value for data quality was used instead of the master id). This makes it impossible to compare the two dumps. Sigh.

It seems that Discogs realized this mistake fairly soon, because on December 15 a new dump with all data from December up until December 12 was released where they corrected this mistake.
So I compared the January dump with this "new December dump", to get at least some of the information that I wanted to get. This "new December dump" has 9,258,776 releases, the January dump has 9,324,867 releases. That means that from December 12 - December 31 66,091 releases more in the database. Of these:
  • 8,916,634 releases stayed the same
  • 340,871 releases were changed
  • 67,362 were added
  • 1,271 were removed from the database
which looks a lot more like what I expected.

Smells

When looking at smells: I cannot really compare, as quite a few of the checks have been improved so more smells are reported and new checks were added. Ignoring incorrect tracklistings (which is by far the most common error in Discogs data) then in December 1,546 releases with possible errors were added. When taking incorrect tracklistings into account that number increases to over 5,400.

When looking at possible errors in all of the data (including the tracklisting errors), then my scripts found a bit over 293,000 releases that are in need of some improvement. These are distributed as follows:

Distribution of possible errors in the Discogs dataset

If the tracklisting errors are not included it looks a bit different:

Distribution of possible errors in the Discogs dataset (no tracklisting errors)


But not all is doom and gloom. Some errors (like the Spanish depósito legal and SPARS Code fields) decreased somewhat and I still have hope that someday the data will be a lot better.

In just a few days a new datadump will be released and this time I have not changed my scripts for checking, so that will make for a more fair comparison.



Labels:

Comments

Post a Comment

Popular posts from this blog

SID codes (part 1)

One thing that I only learned about after using Discogs is the so called Source Identification Code, or SID. These codes were introduced in 1994 to combat piracy and to find out on which machines a CD was made. It was introduced by Philips and adopted by IFPI, and specifications are publicly available which clearly describe the two available SID codes (mastering SID code and mould SID code). Since quite a few months Discogs has two fields available in the " Barcode and Other Identifiers " (BaOI) section: Mould SID code Mastering SID code A few questions immediately popped up in my mind: how many releases don't have a SID field defined when there should be (for example, the free text field indicates it is a SID field)? how many releases have a SID field with values that should not be in the SID field? how many release have a SID field, but a wrong year (as SID codes were only introduced in 1994) how many vinyl releases have a SID code defined (which is impossi
1 comment
Read more

SPARS codes (part 1)

Let's talk about SPARS codes used on CDs (or CD-like formats). You have most likely seen it used, but maybe don't know its name. The SPARS code is a three letter code indicating if recording, mixing and mastering were analogue or digital. For example they could look like the ones below. There is not a fixed format, so there are other variants as well. Personally I am not paying too much attention to these codes (I simply do not care), but in the classical music world if something was labeled as DDD (so everything digital) companies could ask premium prices. That makes it interesting information to mine and unlock, which is something that Discogs does not allow people to do when searching (yet!) even though it could be a helpful filter. I wanted to see if it can be used as an identifier to tell releases apart (are there similar releases where the only difference is the SPARS code?). SPARS code in Discogs Since a few months SPARS is a separate field in the Discogs
Post a Comment
Read more

Country statistics (part 2)

One thing I wondered about: for how many releases is the country field changed? I looked at the two most recent data dumps (covering February and March 2019) and see where they differed. In total 5274 releases "moved". The top 20 moves are: unknown -> US: 454 Germany -> Europe: 319 UK & Europe -> Europe: 217 unknown -> UK: 178 UK -> Europe: 149 Netherlands -> Europe: 147 unknown -> Europe: 139 unknown -> Germany: 120 UK -> US: 118 Europe -> Germany: 84 US -> UK: 79 USA & Canada -> US: 76 US -> Canada: 65 unknown -> France: 64 UK -> UK & Europe: 62 UK & Europe -> UK: 51 France -> Europe: 51 Saudi Arabia -> United Arab Emirates: 49 US -> Europe: 46 unknown -> Japan: 45 When you think about it these all make sense (there was a big consolidation in Europe in the 1980s and releases for multiple countries were made in a single pressing plant) but there are also a few weird changes:
Post a Comment
Read more