Vinyl & Data

I am going to talk about the monthly statistics again, but I am going to skip a few months: that's right, no drill down statistics for January, February and March 2020. The reason is that Discogs had changed the dump format and I first needed to fix my scripts and give them a good brush up. The dump file that I researched contains the data as it was on May 1 2020 (I think as it was at 00:00 UTC, but I am not entirely sure). I compared this to the database from April 1 2020. there were 12,458,766 releases up from 12,302,145 releases, so that's 156,621 more 159,735 releases were added 3,114 releases were removed 11,676,243 releases stayed the same 622,788 releases were modified The modifications in the database were distributed across the data as follows: Releases in Discogs changed in April 2020 As I found out recently there are certain edits that are not very relevant , such as the YouTube links. These are irrelevant for me. The 268,187 releases with only irrelevant edits can...

Time to return to an older subject: ISRC codes. I have written about ISRC codes before so if you don't know what I am talking about, you might want to read the older posts first and then return to this one. I wondered how in the past few months the number of obvious ISRC errors would have gone down. In my previous post about this subject I saw a massive decrease: May 2019 dump: 22,391 releases June 2019 dump: 21,096 releases July 2019 dump: 19,214 releases August 2019 dump: 15,161 releases September 2019 dump: 13,390 releases October 2019 dump: 12,090 releases November 2019 dump: 11,360 releases December 2019 dump: 10,578 releases January 2020 dump: 8,893 releases and I wanted to know if this trend continued: February 2020 dump: 6,840 releases (but this dump covered changes in January and most of February, not just January) March 2020 dump: 6,722 releases April 2020 dump: 6,468 releases May 2020 dump: 6,313 releases The trend is still downwards, but the pace seems to have slowed ...

Recently Discogs changed its data dump format, making it more difficult to compare releases the way I used to do and which is described in a previous blog post . So, the only thing I could do: dive into the XML a bit more and compare elements in the XML output for releases that have been changed. The results are quite interesting. I took the data dumps of March 1 2020 and April 1 2020, split these data dumps into individual XML files, computed SHA256 checksums for each of these files and ignored the ones that were the same in both dataset, leaving me with 531,444 releases to look at: Releases that were changed in March 2020 This graph is the same graph that I have been seeing for the last 2.5 years, but what I wanted to know is: what are the releases with relevant changes and where are those changes? Personally I don't think that the YouTube videos are very relevant, and since that was the biggest change in the Discogs data dumps I decided to filter these and then compare. What I ...

Because Discogs changed the data format of the data dumps and I have not yet fixed the scripts that I use there are no monthly statistics yet. But I can still say something about how many previously correct releases actually had errors introduced. As I said (or at least implied) in the previous article this doesn't necessarily mean that the data of the release is now more wrong: it could be that data was added that wasn't previously there and in the added data there is an error (this is what I mostly see). In January 2020 the 1564 unique releases that had errors introduced were distributed over the data as follows: Releases in the Discogs database where an error was introduced in January 2020 and which were previously correct Now, it should be said that this data actually seems to cover a large part of February 2020 as well. The name of the data dump file seems to suggest that it includes data up to February 20. The 321 unique releases that were changed between February 20 202...

One thing that I am always interested about: how many releases that my scripts thought were OK now have errors? I compared the data of smells of the data dump of December 2019 with the data dump of January 2020 and got the following chart (as always, the columns indicate the range of release numbers in the database: first column is everything with release number < 1,000,000, second column is everything between 1,000,000 and 2,000,000, and so on): Releases in the Discogs database where an error was introduced in January 2020 December 2019 and which were previously correct So actually: not too bad. It is also consistent with the change patterns I have seen over the years. The reasons for the peaks for older releases and newer releases: older releases are expanded with new information. This also means that more errors are introduced there. newer releases are usually added first, and expanded later by other people. This also increases the chances of errors being introduced.

Normally I would have expected this post to have been published earlier, but Discogs didn't release its data earlier. In the past the data dumps would be released around the 4th or 5th of the month, but now it took until February 24th before there was a data dump. I then looked at the data and just threw it into a corner and left it there for a few months. This is going to be a little bit different than the other blog posts in this series . The reason is that the internal format seems to have changed again. In the past the dump file would grow with around 100 MiB per month and it has done this very consistently over years. So when I saw that in one month the archive had grown with 1 GiB (about ten times the expected increase) I knew something was up. A short inspection: a lot of the YouTube video information is now included, so comparing the XML data as I have done so far actually makes no sense at all, as with the new data that is included a release would have been marked as ...

In the last few months I had very little time to look at anything related to Discogs because of work. The fact that Discogs was also very late releasing their monthly data dump in February (and changed the format, which kind of broke my scripts) didn't exactly help either. In the last few days I revisited Discogs and after a few minutes the frustration I have felt in the past immediately returned. Basically it all boils down to the following: people are (still!) allowed to put all kinds of garbage into the database and very often this can be completely prevented, as I have argued before and let Discogs staff know many times over the last (nearly) 3 years. I started doing some minor cleanups and it just feels a bit useless: I have been doing the same clean ups for the last few years, so why do I even bother fixing it if the guardians of the database don't seem to care themselves? A lot of time is now wasted on trivial cleanups which also could have gone to adding data to the...

Today another data dump was released covering all releases up until (and including) December 31 2019, so I can look at some statistics again. If you don't know how this works, I would recommend reading the previous articles . Release statistics I looked at the dump file with new data entered and changes made from December 1 - December 31 2019. This dump file has 11,955,493 releases, whereas the previous one had 11,854,877. That means 100,616 releases more. Also: 11,187,982 releases stayed the same 663,648 releases were changed 103,863 releases were added 3,247 releases were removed from the database 208 releases had status Draft, Deleted or Rejected 0 releases that were not Accepted were in both dumps 3 releases were moved from Draft to Accepted Changes in the data (that is: changes to already existing releases) are distributed as follows: Existing releases changed in December 2019  What is striking is that there are significantly more edits. I looked thr...

Because a new data dump was released today I decided to look a bit further into the ISRC codes and how they are being fixed. Before you read further I would advise to read the previous blog post about this topic first. In the previous article I had the following data for the amount of releases with a (possible) ISRC code, but not using the ISRC field: May 2019 dump: 22,391 releases June 2019 dump: 21,096 releases July 2019 dump: 19,214 releases August 2019 dump: 15,161 releases September 2019 dump: 13,390 releases October 2019 dump: 12,090 releases November 2019 dump: 11,360 releases December 2019 dump: 10,578 releases In the January 2020 dump this is further down to 8,893 releases, so that's good news. I am hopeful that in the next few months that all the obvious mistakes will have been fixed (there are also a few edge cases I need to research further so I can filter them out). To catch that in bar charts: I plotted the release numbers that are in need of fixing f...

When adding a release to Discogs it is important to record as much detail as possible. This means looking at physical releases, discovering what data is there and then entering it. But it is sometimes very difficult to see what actually is on a release. A big challenge is optical media, where some data (such as SID codes ) is sometimes unreadable because it is so small. The result: people are missing information, or enter the wrong information. You basically have no other choice than to magnify. And what better tool to use than a camera? So far I have tried a few things. One thing I tried is a cheap endoscope (originally bought to see deep inside the guts of audio equipment in need of repair), but that is something I would most definitely not recommended, as it is essentially a bad webcam with ditto quality. I also looked at a USB microscope but those are either very expensive, or also webcam quality. Also, I already have a camera that gives me very good quality pictures, namely my...

Having looked through many releases in Discogs I often see that data is not entered as it is on the release. One example: rights societies are typeset in many different ways, where sometimes you will have an all uppper case name (for example: "BIEM"), sometimes there is spacing ("B I E M"), sometimes there are dots in the name ("B.I.E.M."), and so on. In the examples mentioned above the values are all basically referencing the same rights society so it is worth asking if the values entered on each release page should be "as written on release" or if they should be stored as the value of whatever it is pointing it? Said differently: should the release page focus on "syntax" (how it is written on the release) or "semantics" (what it means)? There are things to be said for both: there are releases where the person designing the sleeve or label made an error, for example: releases with "BIEN" instead of "BIEM...

One field in the "barcode and other identifiers" section in Discogs that is fairly new is the ISRC field. I have written about ISRC before, so you might want to read those posts first. Before there was a dedicated ISRC field the ISRC codes would be put into fields like "Other" or "Barcode" so I wondered: how well are these being updated and replaced by ISRC fields? I checked the dumps from the last 6 months: May 2019 dump: 22,391 releases June 2019 dump: 21,096 releases July 2019 dump: 19,214 releases August 2019 dump: 15,161 releases September 2019 dump: 13,390 releases October 2019 dump: 12,090 releases November 2019 dump: 11,360 releases December 2019 dump: 10,578 releases So in 8 months time the number of problematic releases has more than halved, while the amount of releases in the database with ISRC codes has increased. With a bit of luck all of these will have been fixed in the next half year. Impressive.

In some countries (especially in Spain) it was/is custom to translate titles of songs and print them on the sleeve (sometimes without the original title). The cool thing about this is that these releases have unique sleeves and labels. One example is Queen's "Hot Space", which was translated to " Espacio Caliente ". Frequently songs are also covered in a different language than the original and given a new title. From a collector point of view this could be interesting: there are actually people who collect covers of specific songs. The challenge is that if only the translated title is printed on the release (or entered in Discogs) it is harder to find out which track it actually is. For regular releases (where the title simply has been translated) it is usually not that difficult. But, for cover versions it can be. How to indicate translations has been the matter of debate inside Discogs. At some point the solution was to enter both the original title and...

A few days ago another data dump was released covering all releases up until (and including) November 30 2019, so I can look at some statistics again. If you don't know how this works, I would recommend reading the previous articles about it, for example the article about October 2019 . Release statistics I looked at the dump file with new data entered and changes made from November 1 - November 30 2019. This dump file has 11,854,877 releases, whereas the previous one had 11,751,696. That means 103,181 releases more. Also: 11,249,513 releases stayed the same 500,202 releases were changed 105,162 releases were added 1,981 releases were removed from the database 588 releases had status Draft, Deleted or Rejected 0 releases that were not Accepted were in both dumps 3 releases were moved from Draft to Accepted Changes in the data (that is: changes to already existing releases) are distributed as follows: Existing releases changed in November 2019 Smells I fou...

The field in the Discogs database that is definitely the most misunderstood is the so called "Label Code". Many people think that this is a generic "catch all" field for any code printed on a label, but that is COMPLETELY WRONG . The Label Code is a (somewhat) unique identifier for a record label. It was introduced in Europe somewhere in the 1970s and is now in widespread use. This is all explained in the other articles I have written about this field and I recommend you read those first (especially the first article ). Many Discogs contributors think that just because there is a code on a label it should be in the Label Code field. In the latest download (all releases up until October 31 2019) there are a bit over 44,000 releases that have a wrong value in this field. That's quite a few. Then there are also releases where a Label Code should be present, but isn't (and of course I don't know how many), plus around 9,000 where someone indicated that ...

About 1.5 year ago I discovered that some people use the wrong character set in the "Rights Society" field, because characters in various character sets (Latin, Cyrillic, Greek) look alike (the so called "homoglyphs"). Some people even mixed characters from different character sets. You can read the articles I wrote here and here and I would recommend you read those first as I will be making comparisons with the data presented there. I was wondering if this situation had actually changed, or if things got worse, so I ran my scripts and found 1146 releases where I know that the wrong character set(s) were used. The distribution across the data set is as follows: Distribution of releases with using the wrong character set(s) for Rights Society Comparing it to the data from 1.5 years ago it seems to have gotten a little bit worse (though not much). As long as there is no search functionality for rights societies on the Discogs website it won't matter muc...

Sometimes duplicate entries get added to the Discogs database, for various reasons: inexperienced users: it takes some time to understand the Discogs workflow. This happened to me as well when I was starting out and it is because the Discogs edit interface throws you into expert mode, instead of trying to guide you through the process (which I have written about before here and here ). stupid sellers: some sellers still don't understand that Discogs is a catalog where you simply pick a release (unlike for example eBay), so they add releases that are already in the catalog. entry errors: sometimes errors are made, which makes it hard to find that an entry has already been added, for example when it isn't clear what the correct label is, or the artist, and so on and then someone adds the same release twice. disagreement about when a release is a variation: people on Discogs frequently disagree about when a release is actually a different release, or simply a variation. For ...

I haven't looked at SPARS codes for a while but I thought it would be interesting to see which SPARS code is used the most. If you haven't read my previous articles about SPARS codes I suggest you do that first. What I wondered is: how often is each SPARS code used on a release? I looked at the releases where I have verified that there is a valid SPARS code. There are more releases with SPARS codes in Discogs, but a lot of releases (at least 24,000) still need to be fixed at the time of writing and these haven't been looked at. In total I looked at 113,975 releases from data up to October 31 2019. The breakdown: DDD: 52,082 releases AAD: 33,821 releases ADD: 28,564 releases DAD: 480 releases AAA: 120 releases DDA: 87 releases ADA: 15 releases DAA: 12 releases If you add these numbers you will get more than 113,975. This is because some releases have more than one SPARS code as the mix can actually be different per track. What is clear is that the "regul...

Back in 2018 I looked at how many releases had a SPARS code, and a release date before SPARS codes were actually introduced. You can read about that here . Back then I found 30 releases and I promised I would revisit this topic sometime later. I just looked at the data up to October 31 and I found 61 releases. That's actually an increase. Bah. Releases in Discogs with a SPARS code and dated before SPARS codes were introduced It seems there is still some work to do!

I started this blog in September 2017 with the intention of finding interesting patterns in the Discogs data. It fairly quickly turned into a blog pointing out what doesn't work in Discogs, which I must admit is a bit negative and not what I originally wanted. I am honestly trying to keep it positive this simply isn't always possible. The reason: in the last two years very little has been done to get rid of data entry errors that can be prevented. The only thing that I have seen is the automatic correction of capitalization errors (even though it is a bit difficult to trigger, although it does a good job). Due to work obligations I couldn't look at the data for a few months and had hoped that things would have been different when I returned to processing the data, but this wasn't the case. I have had conversations with people at Discogs and those conversations were actually very nice, but it doesn't seem to lead to improvements. Why this is I really don't know...