Vinyl & Data

Recently Discogs changed its data dump format, making it more difficult to compare releases the way I used to do and which is described in a previous blog post . So, the only thing I could do: dive into the XML a bit more and compare elements in the XML output for releases that have been changed. The results are quite interesting. I took the data dumps of March 1 2020 and April 1 2020, split these data dumps into individual XML files, computed SHA256 checksums for each of these files and ignored the ones that were the same in both dataset, leaving me with 531,444 releases to look at: Releases that were changed in March 2020 This graph is the same graph that I have been seeing for the last 2.5 years, but what I wanted to know is: what are the releases with relevant changes and where are those changes? Personally I don't think that the YouTube videos are very relevant, and since that was the biggest change in the Discogs data dumps I decided to filter these and then compare. What I ...

Wow. I just realised that it has been about 9 months since I last analyzed Discogs data. I got swamped by enormous amounts of work so I didn't have time to dive into this subject. Now that things are (temporarily) a bit slower (for a change) it is time to correct that and look into what happened in June 2018. If you don't know how this works, I suggest you read the post about May 2018 and work your way through history. Release statistics The latest dump (which I call "the July dump", as it was released then) covers all data added from June 1 -  June 30 2018 (inclusive). The previous dump had  9,906,032 releases, the new dump has 10,010,587 releases. That means 104,555 releases more in the database. Of those: 9,350,389 releases stayed the same 553,252 releases were changed 106, 946 releases were added 2,391 releases were removed from the database 296 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both dumps 2 rel...

In Discogs each release has one or more Format fields, in which a contributor has to indicate what format a device has, or what formats a device has, in case it has multiple formats. I simply looked at all the releases in the database and simply counted, and this is the list I got (from most releases to fewest releases): Vinyl: 4,819,258 CD: 2,913,785 File: 917,569 Cassette: 647,604 CDr: 386,541 Shellac: 146,494 DVD: 85,475 Box Set: 52,634 All Media: 29,217 Flexi-disc: 20,814 VHS: 18,674 8-Track Cartridge: 15,206 Acetate: 9,580 DVDr: 8,581 Lathe Cut: 8,023 SACD: 6,380 Reel-To-Reel: 5,022 Blu-ray: 3,705 Laserdisc: 3,077 Memory Stick: 1,701 Minidisc: 1,613 Edison Disc: 1,308 Cylinder: 1,290 Betacam SP: 1,060 Hybrid: 1,012 Floppy Disk: 1,003 Blu-ray-R: 674 CDV: 601 4-Track Cartridge: 593 DCC: 397 Pathé Disc: 367 U-matic: 362 Betamax: 263 DAT: 209 PlayTape: 144 Microcassette: 135 HD DVD: 65 MiniDV: 57 UMD: 51 VHD: 40 SelectaVision: 37 Tefifon: 3...

One category of releaes in Discogs are the "digital releases". Basically: MP3s or other digital formats from stores, iTunes releases, and so on. Call me old fashioned, but personally I don't see these as collectables, as to me they are just files on a computer or music player. But apparently many people disagree with that and collect them. In Discogs the digital releases have "File" in the format field. This makes it quite easy to recognize. So I wondered: how many of these "file" releases are there and where are they in the data? Is it mostly the newer releases, or are there also many older file releases? So I looked at everything in the latest data dump and found 917,569 releases that have "File" in the format field, which is about 10% of the releases in Discogs and 40% more than for example cassettes. Distribution of releases tagged as "File" in the Discogs data set So only in the very early days there were few "...

Another month, so another round of statistics. This time Discogs was fairly quick to release a new data dump, unlike last month . If you don't know how this works it is best to first read that overview, plus perhaps a few older ones. Release statistics The latest dump (which I call "the June dump", as it was released then) covers all data added from May 1 May 15 - May 31 2018 (inclusive). The previous dump had 9,843,513 releases, the new dump has 9,906,032 releases. That means 62,519 releases more in the database. Of those: 9,520,889 releases stayed the same 320,855 releases were changed 64,288 releases were added 1,769 releases were removed from the database 365 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both dumps 1 release was moved from Draft to Accepted What strikes me is that the amount of releases is a lot less than last time, although then it was a lot more. So I guess that last month's dump was actua...

I actually had wanted to write this post a few weeks ago, but for some reasons it took Discogs a lot longer to publish the new data dump. I can only guess why, but I would not be surprised if GDPR had something to do with it. I then got swamped with other tasks so now, almost a month too late, I can finally tell you what happened in Discogs in April 2018. First of all, if this is the first time you read one of these posts, please read the one from last month first. Release statistics The latest dump (which I call the "May dump") covers the period of April 1 - April 30 May 14 2018 (inclusive). The previous dump had 9,680,263 releases, the new dump has 9,843,513 releases.  That means 163,250 releases more in the database. Of these: 8,983,224 releases stayed the same 693,583 releases were changed 166,706 releases were added 3,456 releases were removed from the database 162 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both ...

A new month, so that means that there is a new dumpfile available that I can do analysis on. If this is the first time you see one of these posts I would highly recommend to first read the posts from previous months . Release statistics The latest dump ("the April dump") covers the period from March 1 - March 31 (inclusive).  The previous dump had 9,554,069 releases, the new dump has  9,680,263 releases. That means 126,194 releases more in the database. 8,962,136 releases stayed the same 589,131 releases were changed 128,996 releases were added 2,802 releases were removed from the database 247 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both dumps 0 release was moved from Draft to Accepted Luckily this time Discogs did not change the XML format, so it is relatively easy to compare to the previous month. It looks very similar to previous months: the amount of releases edited is very similar, but perhaps with a sl...

Time to look into a few more aspects of the Discogs data model and where I think it could be improved. If you haven't read the first part I would suggest you do that first. This time I want to look at a few more: rights society, mastering and mould SID codes, and pressing plant. Rights Society Currently rights societies are stored in the "Barcodes and Other Identifiers" (BaOI) section which covers the whole release. The correct place for this information would actually be per track. There are some releases in Discogs (like some compilations) where it is indicated which rights society applies to which track. Mastering and Mould SID codes Like rights societies the mastering and mould SID codes are stored in the BaOI section that covers the whole release. The correct place for these identifiers would actually be a physical disc, as they frequently are different in releases with multiple discs per release. The solution that people use is to tag it in the free text fie...

When adding a new artist to Discog and the artist is unknown it is normally added to the database and the release is added to the artist page. Artist actually is a bit of a misnomer, as it could also mean a sound engineer, a producer, and so on. For each of the "artists" there is a number in the database and a corresponding artist page. I was looking through some of the entries in Discogs and spotted a few instances where the number of the artist was 0 and on the release page there was no link to an artist page. The credits list on the Discogs website explains that there are five credits for which the artist is not recorded in the database and there is no artist page. Personally I don't see the need for three of these ("artwork by", "photography" and "executive producer") and think they should be completely replaced by linked credits. I was wondering how many releases there actually are where one or more "artists" were not li...

After looking at many releases in Discogs and the edit history of various releases and how things have changed through time it is quite clear to see how the data model has changed. Some of these changes have turned out well, while others were, in retrospect, probably not the right change to make. I am not blaming the developers as I know from experience how difficult it is to get a data model right the first time and how hard it is to change: as soon as something is in use it is non-trivial to change especially with a database the size of Discogs. Still I want to go through a few examples where I think the current datamodel doesn't work. Most of my examples will focus on the fields from the "Barcodes and Other Identifiers" (BaOI) section. In this post I am looking at three of them: ISRC, SPARS codes and matrix/runout. ISRC Currently ISRC codes (International Standard Recording Code, see previous articles about it for more information: 1 , 2 , 3 ) are stored in the ...

It is a new month, so that means that there is a new dumpfile available that I can do analysis on. If this is the first time you see one of these posts I would highly recommend to first read a similar post from last month . Release statistics The latest dump ("the March dump") contains data from February 1 - February 28 (inclusive).  The previous dump had 9,442,719 releases, the new dump has  9,554,069 releases. That means 111,350 releases more in the database. 3,391,491 releases stayed the same 6,048,861 releases were changed 113,717 releases were added 2,367 releases were removed from the database 209 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both dumps 1 release was moved from Draft to Accepted So, looking at the releases that were changed it is obvious: Discogs changed the dump format again! ARGH! This time it is in the "joiners" for the artist names. Discogs changing the internal XML format makes it...

Using pressing plant information to find wrong information in releases in Discogs is something that I explored in an earlier post . I only looked at a single pressing plant and that already uncovered 76 releases that were obviously incorrect. I added a few more checks for pressing plants to my scripts and processed the latest Dicogs data dump that I could find. Immediately hundreds of more errors popped up, and I didn't even add that many companies to my checks. With the new checks I could find 610 that are wrong, but I am sure that as soon as I start adding more checks many more errors will pop up. Releases in Discogs where manufacturing plants and release years don't match What often seems to be the case that people have combined the original release with reissues, making the entries completely useless for a correct classification. One thing that I noticed when looking at the pages for the manufacturing plants is that there are quite a few where it is mentioned w...

Something that has turned out surprisingly hard to do is to categorize CDs. While at first it seemed easier than vinyl it has turned out to be much more difficult, as CDs have been repressed in different years by different plants (or the same under a different name), but where everything is the same, except for some very minor details, such as SID codes, or the CD matrix. I have already written about SID codes and how they can be used to very roughly date CD releases . But there is also other information that can be used to see if releases have been dated (somewhat) correctly. Recently I decided to look at the companies that are listed, see when they were operational and use that information for checking release years. Quite a few CDs were pressed at a particular plant in the US. During its lifetime this plant operated under different names: PDO, USA from 1986 - 1992 PMDC, USA from 1992 - 1999 UML from 1999 - 2005 EDC, USA from 2005 - 2009 There is some overlap: some g...

A new month, so Discogs uploaded a new datadump that I could analyze. Before reading this blogpost it might be good to first read the one about last month's datadump . Release statistics The latest dump ("the February dump") contains data from January 1 - January 31 (inclusive).  The previous dump had 9,324,867 releases, the new dump has 9,442,719 releases. That means 117,852 releases more in the database which seems to be a bit more than in previous months. 8,776,440 releases stayed the same 545,810 releases were changed 120,469 releases were added 2,617 releases were removed from the database 179 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both dumps 1 release was moved from Draft to Accepted All in all it looks like a very typical month in Discogs. Smells When looking at absolute numbers of releases for which I have identified a possible smell there is good news: 1,236 releases with possible known smells les...

A bit later than usual (I was swamped with work), but there is a new datadump, so it is time to dive into statistics again. It might be good to read one of the previous posts before reading this one. Release statistics I downloaded the latest database dump (the "January dump") containing data from December 1 to December 31 (inclusive). The previous dump had 9,217,123 releases, the new dump has 9,324,867 releases. That means 107,744 releases more in the database, which seems to be quite consistent with the previous months. 3,916,654 releases stayed the same 5,298,409 releases were changed 109,804 were added 2,060 releases were removed from the database 128 releases had status Draft , Deleted or Rejected 11 releases that were not Accepted were in both the December and January dumps 2 releases were moved from Draft to Accepted So what is immediately obvious is that like with the December dump there is again a very large amount of changed releases and like last t...

After having seen quite a few releases on Discogs and having dug through the data that Discogs makes available every month I am sometimes still finding things that make me wonder. In a previous post I already talked a bit about what I could do if the whole edit history would be made available, but I forgot one important thing: some people tend to put information that is relevant to the release in the submission notes. The submission notes are just metadata describing an edit . They should not be used to describe a release . Yet some people do exactly this. They do not realize that unless you are logged into Discogs and actively click on "edit page" you will not see the submission history and you will not see that particular information. It is effectively hidden for people who do not know this or who are not logged in. To find releases with that information it would be really useful to have the Discogs edit history. On the other hand, maybe it should also be a signal to...

Discogs is making its database data available under a CC0 license, but one important aspect of the data is lacking and that is the edit history. For me as a software engineer having the edit history available is very natural, as it allows me to dive into the history perhaps see where people went wrong, when certain errors were introduced, and so on. For Discogs this is only possible if I am logged into the website and go to the release page history where I am presented with a view of the edit history without being able to interact with it (queries, and so on), which I find frustrating. The edit history is a very rich place of information. For each release a lot of information is kept, such as: creation date/modifcation date origin (whether it was created fresh, or if "copy to draft" was used) merge information (if merged with another release) content of edits contributors and votes While some of this information is partially available through the Discogs API the ful...

A new month, a new data dump...and new statistics! Before reading this post it might be good to read the post about what happened in Discogs in October 2017 . Release statistics For this blogpost I downloaded the latest datadump (the "December dump") containing data from November 1 to November 30 (inclusive). The previous dump file had 9,107,428 releases, the new dump file has 9,217,123 releases. That means 109,695 more releases in the database. 3,868,975 releases stayed the same 5,235,697 releases were changed 112,451 releases were added 2,756 releases were removed from the database 222 releases had status Draft , Deleted or Rejected 11 releases that were not Accepted were in both the November dump and the December dump  2 releases were moved from Draft to Accepted What immediately stands out is that there is an enormous amount of releases that has been changed compared to previous months, when it was about 10 times less . As I don't believe that Discog...

I just spent half an hour listening to a friend who has been cleaning up entries where a user entered a rights society, but used the wrong field. What he often saw is an Amazon identifier ( ASIN ), even though it was very clearly not (Amazon identifiers are well defined ). Of course, what happened here is that (before the last big BaOI update) the ASIN and Rights Society choices in the BaOI drop down box were directly next to eachother and people simply chose the wrong one. Some of these mistakes were in Discogs for several years. These errors would have been very easy to detect and prevent. Errors that have been in Discogs for years could have been very easily prevented. A few possible checks that could have been implemented: check if the value of an ASIN field matches the ASIN specification check if the value of a field (if not Rights Society ) happens to be a known rights society, or even a subset of righst societies (like BIEM, with or without dots) This is not impossib...

One cool thing about Discogs is that you can see the whole history of the data, including who edited when and what was changed. For a software engineer like me that is very convenient, as I am used to working with version control. Unfortunately this data is missing from the Discogs datadump, so I can not easily see which data has changed compared to a previous month. This makes it a bit more difficult to for example see if things are being changed incorrectly, or spot other problems. Having a field with a timestamp of the last change to the release would be great to have. I suggested it to Discogs, but I am not sure if they will add it to the XML, or maybe they will save it for a next update of the format. In any case, I wanted to be able to see the differences between months now , so I came up with an awful hack, but one that would also make it possible for me to do some very crude filtering, as well as find differences between any two months, and not just consecutive months, at t...