Skip to main content

Digging through the Discogs XML

At the beginning of every month (usually the 3rd or 4th day of the month) Discogs releases a data dump of various parts of the database:
  • individual releases
  • master releases
  • artists
  • labels
This data is released under the CC0 license, basically allowing unlimited reuse, which is very cool and opens up all kinds of possibilities, which got me really excited about the possibilities (and caused a sleepless night or two).

I have not fully looked into all the data and in this post (and the next few) I will only look at the XML of the individual releases.

Individual releases in the Discogs data

Every month the data for all individual releases of Discogs is made available as a gzip compressed XML dump. This file is quite big: the dump for September 2017 is 4.9 GiB when it is gzip compressed. Uncompressed it is 32 GiB. The data dump for September 2017 contains information about 8,878,391 releases
The Discogs database had information about  8,878,391 releases on September 1 2017
The top level element of the file is called releases. All the individual releases are children of this top level element. Each release is contained in an element release. This element has a few important attributes:
  • id : the identifier in the database
  • status : the status of the release in the database. Currently a release can be accepted (8,873,625 in September 2017 dump), rejected (1,442 in September 2017 dump), deleted (1,005 in September 2017 dump) or draft (2,319 in September 2017 dump).
Each release element has a few children:
  • images - metadata about the images of the release uploaded to Discogs (not included in the data dumps, only available via the API)
  • artists - artists on the release
  • title - the title of the release
  • labels - music labels involved in the release
  • extraartists - artist data of any guest artists
  • formats - formats of the release (usually one, but there can be multi-format releases)
  • genres - genres the release fits in
  • styles - musical styles
  • country - country (or meta-country like EEC) of the release
  • released - date (year, or full date, or empty) for the release
  • notes - free text field for data that didn't fit anywhere else
  • master_id - the name of the master release (data of which can be found in another XML file)
  • data_quality - quality of the release data as voted for by Discogs users with voting rights
  • tracklist - list of tracks on the releases
  • identifiers - matrix/runouts, barcodes, label codes, and so on
  • videos - links to (external) video sites with video clips of songs on the release
  • companies - companies involved in the release (pressing, mastering, and so on)

Which XML parser to use?

Because of its size you don't want to process the XML with a DOM parser. For processing a file of this size a SAX parser is a much better option.
Use a SAX parser for processing the Discogs XML, not a DOM parser.
Since the XML is generated from the Discogs database it always has the same format and the (main) elements always appear in the same order, allowing for a few shortcuts. However, if you want to cross reference data from various fields you will have to process it in another way, for example first splitting the XML into smaller pieces (exaple: 10,000) and then processing each chunk separately (for example using xml_split), parsing the data with a SAX parser and building an internal data representation and subsequently using that data or possibly storing it in a database first.

Discogs error detection script

I have been working on a script to cleanup the data, which I made available under the GPL 3 license. The script can be found in a repository on GitHub. You will need Python3 to run it. I have only tested it on Linux, but it should work on other platforms as well.

The script take the XML, processes it and reports any smells. Currently it only looks at a few of the available fields (namely identifiers, notes, country) and performs some very simplistic checks to see if the data in the fields in identifiers make sense or if the wrong value is used, or if it is in the wrong place (wrong identifier, or hidden in notes and not added to identifiers). As the script is still under active development more checks will be added.

In the next few blog posts I will walk through each of these checks, plus provide some background information about each of the values and the checks with, hopefully, some interesting statistics.
Labels:

Comments

Post a Comment

Popular posts from this blog

SID codes (part 1)

One thing that I only learned about after using Discogs is the so called Source Identification Code, or SID. These codes were introduced in 1994 to combat piracy and to find out on which machines a CD was made. It was introduced by Philips and adopted by IFPI, and specifications are publicly available which clearly describe the two available SID codes (mastering SID code and mould SID code). Since quite a few months Discogs has two fields available in the " Barcode and Other Identifiers " (BaOI) section: Mould SID code Mastering SID code A few questions immediately popped up in my mind: how many releases don't have a SID field defined when there should be (for example, the free text field indicates it is a SID field)? how many releases have a SID field with values that should not be in the SID field? how many release have a SID field, but a wrong year (as SID codes were only introduced in 1994) how many vinyl releases have a SID code defined (which is impossi...
1 comment
Read more

SPARS codes (part 1)

Let's talk about SPARS codes used on CDs (or CD-like formats). You have most likely seen it used, but maybe don't know its name. The SPARS code is a three letter code indicating if recording, mixing and mastering were analogue or digital. For example they could look like the ones below. There is not a fixed format, so there are other variants as well. Personally I am not paying too much attention to these codes (I simply do not care), but in the classical music world if something was labeled as DDD (so everything digital) companies could ask premium prices. That makes it interesting information to mine and unlock, which is something that Discogs does not allow people to do when searching (yet!) even though it could be a helpful filter. I wanted to see if it can be used as an identifier to tell releases apart (are there similar releases where the only difference is the SPARS code?). SPARS code in Discogs Since a few months SPARS is a separate field in the Discogs ...
Post a Comment
Read more

Country statistics (part 2)

One thing I wondered about: for how many releases is the country field changed? I looked at the two most recent data dumps (covering February and March 2019) and see where they differed. In total 5274 releases "moved". The top 20 moves are: unknown -> US: 454 Germany -> Europe: 319 UK & Europe -> Europe: 217 unknown -> UK: 178 UK -> Europe: 149 Netherlands -> Europe: 147 unknown -> Europe: 139 unknown -> Germany: 120 UK -> US: 118 Europe -> Germany: 84 US -> UK: 79 USA & Canada -> US: 76 US -> Canada: 65 unknown -> France: 64 UK -> UK & Europe: 62 UK & Europe -> UK: 51 France -> Europe: 51 Saudi Arabia -> United Arab Emirates: 49 US -> Europe: 46 unknown -> Japan: 45 When you think about it these all make sense (there was a big consolidation in Europe in the 1980s and releases for multiple countries were made in a single pressing plant) but there are also a few weird changes:...
Post a Comment
Read more