Vinyl & Data

I am going to talk about the monthly statistics again, but I am going to skip a few months: that's right, no drill down statistics for January, February and March 2020. The reason is that Discogs had changed the dump format and I first needed to fix my scripts and give them a good brush up. The dump file that I researched contains the data as it was on May 1 2020 (I think as it was at 00:00 UTC, but I am not entirely sure). I compared this to the database from April 1 2020. there were 12,458,766 releases up from 12,302,145 releases, so that's 156,621 more 159,735 releases were added 3,114 releases were removed 11,676,243 releases stayed the same 622,788 releases were modified The modifications in the database were distributed across the data as follows: Releases in Discogs changed in April 2020 As I found out recently there are certain edits that are not very relevant , such as the YouTube links. These are irrelevant for me. The 268,187 releases with only irrelevant edits can...

Time to return to an older subject: ISRC codes. I have written about ISRC codes before so if you don't know what I am talking about, you might want to read the older posts first and then return to this one. I wondered how in the past few months the number of obvious ISRC errors would have gone down. In my previous post about this subject I saw a massive decrease: May 2019 dump: 22,391 releases June 2019 dump: 21,096 releases July 2019 dump: 19,214 releases August 2019 dump: 15,161 releases September 2019 dump: 13,390 releases October 2019 dump: 12,090 releases November 2019 dump: 11,360 releases December 2019 dump: 10,578 releases January 2020 dump: 8,893 releases and I wanted to know if this trend continued: February 2020 dump: 6,840 releases (but this dump covered changes in January and most of February, not just January) March 2020 dump: 6,722 releases April 2020 dump: 6,468 releases May 2020 dump: 6,313 releases The trend is still downwards, but the pace seems to have slowed ...

Recently Discogs changed its data dump format, making it more difficult to compare releases the way I used to do and which is described in a previous blog post . So, the only thing I could do: dive into the XML a bit more and compare elements in the XML output for releases that have been changed. The results are quite interesting. I took the data dumps of March 1 2020 and April 1 2020, split these data dumps into individual XML files, computed SHA256 checksums for each of these files and ignored the ones that were the same in both dataset, leaving me with 531,444 releases to look at: Releases that were changed in March 2020 This graph is the same graph that I have been seeing for the last 2.5 years, but what I wanted to know is: what are the releases with relevant changes and where are those changes? Personally I don't think that the YouTube videos are very relevant, and since that was the biggest change in the Discogs data dumps I decided to filter these and then compare. What I ...

Because Discogs changed the data format of the data dumps and I have not yet fixed the scripts that I use there are no monthly statistics yet. But I can still say something about how many previously correct releases actually had errors introduced. As I said (or at least implied) in the previous article this doesn't necessarily mean that the data of the release is now more wrong: it could be that data was added that wasn't previously there and in the added data there is an error (this is what I mostly see). In January 2020 the 1564 unique releases that had errors introduced were distributed over the data as follows: Releases in the Discogs database where an error was introduced in January 2020 and which were previously correct Now, it should be said that this data actually seems to cover a large part of February 2020 as well. The name of the data dump file seems to suggest that it includes data up to February 20. The 321 unique releases that were changed between February 20 202...

One thing that I am always interested about: how many releases that my scripts thought were OK now have errors? I compared the data of smells of the data dump of December 2019 with the data dump of January 2020 and got the following chart (as always, the columns indicate the range of release numbers in the database: first column is everything with release number < 1,000,000, second column is everything between 1,000,000 and 2,000,000, and so on): Releases in the Discogs database where an error was introduced in January 2020 December 2019 and which were previously correct So actually: not too bad. It is also consistent with the change patterns I have seen over the years. The reasons for the peaks for older releases and newer releases: older releases are expanded with new information. This also means that more errors are introduced there. newer releases are usually added first, and expanded later by other people. This also increases the chances of errors being introduced.

Normally I would have expected this post to have been published earlier, but Discogs didn't release its data earlier. In the past the data dumps would be released around the 4th or 5th of the month, but now it took until February 24th before there was a data dump. I then looked at the data and just threw it into a corner and left it there for a few months. This is going to be a little bit different than the other blog posts in this series . The reason is that the internal format seems to have changed again. In the past the dump file would grow with around 100 MiB per month and it has done this very consistently over years. So when I saw that in one month the archive had grown with 1 GiB (about ten times the expected increase) I knew something was up. A short inspection: a lot of the YouTube video information is now included, so comparing the XML data as I have done so far actually makes no sense at all, as with the new data that is included a release would have been marked as ...

Today another data dump was released covering all releases up until (and including) December 31 2019, so I can look at some statistics again. If you don't know how this works, I would recommend reading the previous articles . Release statistics I looked at the dump file with new data entered and changes made from December 1 - December 31 2019. This dump file has 11,955,493 releases, whereas the previous one had 11,854,877. That means 100,616 releases more. Also: 11,187,982 releases stayed the same 663,648 releases were changed 103,863 releases were added 3,247 releases were removed from the database 208 releases had status Draft, Deleted or Rejected 0 releases that were not Accepted were in both dumps 3 releases were moved from Draft to Accepted Changes in the data (that is: changes to already existing releases) are distributed as follows: Existing releases changed in December 2019  What is striking is that there are significantly more edits. I looked thr...

Because a new data dump was released today I decided to look a bit further into the ISRC codes and how they are being fixed. Before you read further I would advise to read the previous blog post about this topic first. In the previous article I had the following data for the amount of releases with a (possible) ISRC code, but not using the ISRC field: May 2019 dump: 22,391 releases June 2019 dump: 21,096 releases July 2019 dump: 19,214 releases August 2019 dump: 15,161 releases September 2019 dump: 13,390 releases October 2019 dump: 12,090 releases November 2019 dump: 11,360 releases December 2019 dump: 10,578 releases In the January 2020 dump this is further down to 8,893 releases, so that's good news. I am hopeful that in the next few months that all the obvious mistakes will have been fixed (there are also a few edge cases I need to research further so I can filter them out). To catch that in bar charts: I plotted the release numbers that are in need of fixing f...

When adding a release to Discogs it is important to record as much detail as possible. This means looking at physical releases, discovering what data is there and then entering it. But it is sometimes very difficult to see what actually is on a release. A big challenge is optical media, where some data (such as SID codes ) is sometimes unreadable because it is so small. The result: people are missing information, or enter the wrong information. You basically have no other choice than to magnify. And what better tool to use than a camera? So far I have tried a few things. One thing I tried is a cheap endoscope (originally bought to see deep inside the guts of audio equipment in need of repair), but that is something I would most definitely not recommended, as it is essentially a bad webcam with ditto quality. I also looked at a USB microscope but those are either very expensive, or also webcam quality. Also, I already have a camera that gives me very good quality pictures, namely my...

One field in the "barcode and other identifiers" section in Discogs that is fairly new is the ISRC field. I have written about ISRC before, so you might want to read those posts first. Before there was a dedicated ISRC field the ISRC codes would be put into fields like "Other" or "Barcode" so I wondered: how well are these being updated and replaced by ISRC fields? I checked the dumps from the last 6 months: May 2019 dump: 22,391 releases June 2019 dump: 21,096 releases July 2019 dump: 19,214 releases August 2019 dump: 15,161 releases September 2019 dump: 13,390 releases October 2019 dump: 12,090 releases November 2019 dump: 11,360 releases December 2019 dump: 10,578 releases So in 8 months time the number of problematic releases has more than halved, while the amount of releases in the database with ISRC codes has increased. With a bit of luck all of these will have been fixed in the next half year. Impressive.

A few days ago another data dump was released covering all releases up until (and including) November 30 2019, so I can look at some statistics again. If you don't know how this works, I would recommend reading the previous articles about it, for example the article about October 2019 . Release statistics I looked at the dump file with new data entered and changes made from November 1 - November 30 2019. This dump file has 11,854,877 releases, whereas the previous one had 11,751,696. That means 103,181 releases more. Also: 11,249,513 releases stayed the same 500,202 releases were changed 105,162 releases were added 1,981 releases were removed from the database 588 releases had status Draft, Deleted or Rejected 0 releases that were not Accepted were in both dumps 3 releases were moved from Draft to Accepted Changes in the data (that is: changes to already existing releases) are distributed as follows: Existing releases changed in November 2019 Smells I fou...

The field in the Discogs database that is definitely the most misunderstood is the so called "Label Code". Many people think that this is a generic "catch all" field for any code printed on a label, but that is COMPLETELY WRONG . The Label Code is a (somewhat) unique identifier for a record label. It was introduced in Europe somewhere in the 1970s and is now in widespread use. This is all explained in the other articles I have written about this field and I recommend you read those first (especially the first article ). Many Discogs contributors think that just because there is a code on a label it should be in the Label Code field. In the latest download (all releases up until October 31 2019) there are a bit over 44,000 releases that have a wrong value in this field. That's quite a few. Then there are also releases where a Label Code should be present, but isn't (and of course I don't know how many), plus around 9,000 where someone indicated that ...

About 1.5 year ago I discovered that some people use the wrong character set in the "Rights Society" field, because characters in various character sets (Latin, Cyrillic, Greek) look alike (the so called "homoglyphs"). Some people even mixed characters from different character sets. You can read the articles I wrote here and here and I would recommend you read those first as I will be making comparisons with the data presented there. I was wondering if this situation had actually changed, or if things got worse, so I ran my scripts and found 1146 releases where I know that the wrong character set(s) were used. The distribution across the data set is as follows: Distribution of releases with using the wrong character set(s) for Rights Society Comparing it to the data from 1.5 years ago it seems to have gotten a little bit worse (though not much). As long as there is no search functionality for rights societies on the Discogs website it won't matter muc...

I haven't looked at SPARS codes for a while but I thought it would be interesting to see which SPARS code is used the most. If you haven't read my previous articles about SPARS codes I suggest you do that first. What I wondered is: how often is each SPARS code used on a release? I looked at the releases where I have verified that there is a valid SPARS code. There are more releases with SPARS codes in Discogs, but a lot of releases (at least 24,000) still need to be fixed at the time of writing and these haven't been looked at. In total I looked at 113,975 releases from data up to October 31 2019. The breakdown: DDD: 52,082 releases AAD: 33,821 releases ADD: 28,564 releases DAD: 480 releases AAA: 120 releases DDA: 87 releases ADA: 15 releases DAA: 12 releases If you add these numbers you will get more than 113,975. This is because some releases have more than one SPARS code as the mix can actually be different per track. What is clear is that the "regul...

Back in 2018 I looked at how many releases had a SPARS code, and a release date before SPARS codes were actually introduced. You can read about that here . Back then I found 30 releases and I promised I would revisit this topic sometime later. I just looked at the data up to October 31 and I found 61 releases. That's actually an increase. Bah. Releases in Discogs with a SPARS code and dated before SPARS codes were introduced It seems there is still some work to do!

One thing I keep asking myself is how big changes in Discogs releases typically are: are the changes just small tweaks, or are they really big changes? I decided to compare the size of the changes of releases in two months using a very simple method: looking at the size of the differences. I decided to not go for a very advanced approach that would require me to parse the XML and compare each field, but instead go fora much simpler approach and look at the size differences of the raw XML data. I took two data dumps from the Discogs data website (namely discogs_20191003_releases.xml.gz and discogs_20191101_releases.xml.gz). I split these two dumps into individual XML files (one per release) and renamed them to have the right release number in the file name. I then compared the files with a regular cryptographic hash (SHA256) to see which files were different. These files (439,651 in total) were compared using a locality sensitive hash, specifically TLSH as I am quite familiar wi...

Phew, I finally managed to process all the data that I wanted to and I should be up to date, at least for the time being. So, let's look at what happened in Discogs in October 2019! For those of you who already know what to expect: it is the same as in previous months. For those of you who don't know, please read the post about September 2019 and work your way back . Release statistics I looked at the dump file with data covering October 3 - October 31 2019 (the previous datadump was for a little bit more than September). This dump file has 11,751,696 releases, whereas the previous one had 11,653,094 releases. That means 98,602 releases more which is less than normal, but also keep in mind that this dump file covers fewer days. Also: 11,211,165 releases stayed the same 439,651 releases were changed 100,880 releases were added 2,278 releases were removed from the database 223 releases had status Draft, Deleted or Rejected 0 releases that were not Accepted were in ...

Normally I don't think too much about statistics of a particular month, but September is different, because of Discogs' "September Pledge Initiative" or "S.P.IN". You can read Discogs' own take on it on the Discogs blog , but let's see what the data says. If you don't know what everything here means, just look at the previous blog post and work your way back. Release statistics I looked at the dump file with data covering September 1 - October 3 2019 (according to the name of the datadump). This dump file has 11,653,094 releases, whereas the previous one had 11,530,540 releases. That means 122,554 releases more which is actually not more than in a normal month (at least, that's what I think). Also: 11,021,531 releases stayed the same 506,609 releases were changed 124,954 releases were added 2,400 releases were removed from the database 477 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in...

Time to dig into some statistics about how Discogs was changed in August 2019. Did the excessive heat reduce the number of contributions, or was it business as usual? If you haven't read it yet I would recommend reading the blog post about contributions in July 2019 first. Release statistics I looked at the dump file with data covering August 1 - 31 2019. This dump file has 11,530,540 releases, whereas the previous one had 11,430,638 releases. That means 99,920 releases more which is only a little bit less than normal. Also: 10,897,333 releases stayed the same 529,657 releases were changed 103,550 releases were added 3,648 releases were removed from the database 403 releases had status Draft, Deleted or Rejected 11 releases that were not Accepted were in both dumps 3 releases were moved from Draft to Accepted What is interesting: quite a few more releases were removed from the database, so my guess is that there was some cleanup of sorts. Changes in the data (t...

It is already November, so it is time to write an update about July. I couldn't do it earlier because I got completely swamped in work, but I will be making a few extra blog posts to compensate for the lack of blog posts. Anyway, back to the monthly statistics! If you don't know how this works, you should read the previous entries, for example about what happened in June 2019 and then work your way through history. You will notice certain patterns. Release statistics I looked at the dump file with data covering July 1 - 31 2019. This dump file has 11,430,638 releases, whereas the previous one had 11,334,008 releases. That means 96,630 releases more which is a bit less than normal, but that can probably be explained by summer on the northern hemisphere. Also: 10,853,019 releases stayed the same 478,710 releases were changed 98,909 releases were added 2,279 releases were removed from the database 344 releases had status Draft, Deleted or Rejected 0 11 releases t...