Vinyl & Data

About 1.5 year ago I discovered that some people use the wrong character set in the "Rights Society" field, because characters in various character sets (Latin, Cyrillic, Greek) look alike (the so called "homoglyphs"). Some people even mixed characters from different character sets. You can read the articles I wrote here and here and I would recommend you read those first as I will be making comparisons with the data presented there. I was wondering if this situation had actually changed, or if things got worse, so I ran my scripts and found 1146 releases where I know that the wrong character set(s) were used. The distribution across the data set is as follows: Distribution of releases with using the wrong character set(s) for Rights Society Comparing it to the data from 1.5 years ago it seems to have gotten a little bit worse (though not much). As long as there is no search functionality for rights societies on the Discogs website it won't matter muc...

I just need to rant again about something in Discogs that I really do not like and that is the voting system. In Discogs you can vote on the quality of the data of a release. For example, if someone completely screwed up an existing release, you can vote that it is "Entirely Incorrect" and that will then revert the commit and restore it to the previous state. Or, you could vote that it is "Complete and Correct", signalling that all the data that can be on there is on there and it is correct as well. There are also: Correct - not all information might be there, but at least it is correct Needs Minor Changes - information is there, just needs a quick brush up Needs Major Changes - the release needs a lot of work This is to have some sort of "self cleansing" mechanism, to alert people to mistakes, and to weed out bad contributors, as getting many bad votes will put you into "Discogs school" (the "Contrinbutor Improvement Program", o...

Whenever a change is made in Discogs a mail is sent. In my opinion the system behind these e-mails is broken, as I have already said before . One of the biggest issues that I have with the current e-mails being sent by Discogs is the subject line of the e-mail, saying: Recent changes affecting releases in your collection or Recent changes affecting your contributions but nothing else. I first have to click on the message to see to full content. It might be that I find some of the changes more interesting than the others. For example, if images are changed, then it is likelier that I will want to take a look because sometimes people change images, with ugly consequences . So I would like to have a little bit more information in the e-mail subject, for example: [IMAGES] Recent changes affecting releases in your collection would already be an improvement over the existing mails being sent as it would allow me to prioritize which mails to check first (at the expense of some e...

I am going through a few hundred 7" to see if they are already on Discogs (which in itself is already a big task) and I am seeing quite a few that are not on there (so setting them apart to make scans, which is even more work). What I see way too often is so called "release hijacking" where an existing release is changed into a different release. This seriously pisses me off and I have written about it before . Some examples: someone posts pictures saying "alternate labels" (which according to the guidelines should be turned into a different release) or "better pictures" which turn out to be for a different release altogether (different texts, labels, etc.). Or, they don't read the notes that are for a release that say something like "This is the release with X, for the release with Y, see Z" and then add pictures for the release with Y. Sigh. What these people do not seem to realise is this vastly reduces the value of the database f...

If you search the Discogs forums you will sometimes see posts from people who dream of giving up their regular job and make their money with Discogs and wonder if that is a smart idea. I was asking myself the same question, not because I want to become a full time record dealer, but because I wondered if it actually is possible. After researching it a bit my answer is: most likely it is not possible, unless you are willing to put in a lot of time and effort, accept a cut in income, add uncertainty and perhaps move to another country to cut costs. I live in the Netherlands, which is definitely one of the more expensive countries to live in in Europe: incomes are higher than in most parts of Europe, but taxes are also higher, as are prices for many things, such as housing, restaurants, postage, flights (dynamic pricing, sigh), and so on. So that is already a bit of a challenge. According to the Dutch government the minimum wage (from July 1 2019 onwards) is € 1635.60 per month (bef...

Some people, including me, use Discogs for having an overview which releases they have. Though it is not perfect because people can, and do, change releases so it matches their own copy (even when it should have been a separate release), but it is better than nothing. There are a few things that annoy me in the way that Discogs organizes collections. One of them is the way that releases can be organized into folders. I understand the rationale behind it: you can organize releases into folders and then look at these folders separately instead of at a big list which would work for a few items, but not when having thousands or tens of thousands of items. Except: for me it is too restrictive. Sometimes I would like to have releases in more than one folder, for example I might want have a release in a folder "death metal", a folder "dutch", a folder "coloured vinyl" and a folder "football" at the same time (although I am not aware of any football ...

I have been thinking a bit about how to increase data quality when entering data. There is already a lot that could be done by using wizards and asking users the right questions . But I think that it can be made even easier when using graphical hints and explicitly pointing out to users what information should be entered in the database. In the previous article I mentioned using a wizard and guiding users through the process of entering data. When the right questions have been answered (such as country and label) the user could be asked what the label of the release looks like . For example they could be given the following choice (left: typical label EMI used in Spain in the 1980s, right: typical label EMI used in Spain in the 1970s): Examples of labels EMI used in the 1980s (left) and 1970s (right) Based on this they could then be guided through the process of picking the right data. Also, using which label was picked already means that some checks can be applied. For exa...

I just helped someone add a few releases to Discogs and it was, again, a quite frustrating experience, even though I have added releases to Discogs before. Adding releases to Discogs is actually a lot of work. These are the steps I typically take for a 7" single: see if the release is already there on Discogs. see if the release I have differs from any of the listed releases copy an existing release to draft, or start from scratch fill in all the details that I am sure about make scans, crop the scans, scale them add scans Adding a release properly, with all the details, can easily take 15 to 20 minutes, and then I still only have a fairly barebones release. What I typically don't get right due to lack of knowledge are things like: composers printing companies manufacturing companies pressing plants country specific peculiarities etc. These usually require specialist knowledge about releases from a certain country, label or artist, which I simply don't...

I think it is time for another usability rant. Although I myself am far from a usability expert (the best courses I took at university were about usability that taught me to stay far away from it) I do suffer from user interfaces with quirks and Discogs is no exception. I have written about this subject before ( images , data entry , etc.) but it bites me and my friend gerjolp every time we are on Discogs. While editing releases for correctness (such as the depósito legal fixes described an an earlier post ) where people made snarky remarks about whether or not we didn't have anything better to do. This was a bit strange, as the data for the releases is more complete now and this should be encouraged. My guess is that the real reason is not that the releases were edited but the fact that Discogs sends a notification e-mail for every change that is made and some people got hundreds of e-mails with all an identical message about a similar change in many releases. Instead of flagg...

I rant a lot about Discogs, but it is definitely not all bad (far from it). One of the things that my scripts catch are wrong characters in Czech and Czechoslovak releases. There is a whole forum thread about that where we put findings of my scripts so people can fix the errors. What was very cool is that someone spent quite some time to fix most of the entries my scripts found. This is exactly what I wanted to achieve: empower people so they can help fix the database in a very targeted way, and get better data for everyone. It certainly has given me a boost to look for more easy fixes that people can apply. So pushkin_cz: thank you!

I have been digging into the Discogs dataset for over 9 months now and blogging about it since September. During this period I have made a few observations. In short I must say that I have mixed feelings about Discogs because it isn't clear what Discogs actually is and what they want to achieve. I talked about this earlier , but it basically comes down to this: Is it a catalog? Is it a marketplace? Is it a place to organize your collection? Discogs is trying to be all but not succeeding because there are tensions between the different use cases: enforcing correctness chases sellers away from the marketplace and selling records is what is bringing in the money for Discogs and which keeps the other fires burning. But the costs for this is that the data in the catalog is sometimes blatantly incorrect, which vastly reduces its value. Personally I don't care about the marketplace, as I am a collector and I care about the catalog. Discogs has enormous potential and for collecto...

What I like about working with the Discogs data is to make data that isn't visible visible. In an earlier post I talked about that I suspected that the Discogs contributor ranking likely followed the 80/20 rule , but I didn't have enough data yet to confirm that. I crawled more data from Discogs (very slowly, as Discogs doesn't make it easy with their anti-crawling measure, so I crawled from multiple locations over quite a few hours) and reran scripts that I wrote to crunch the numbers and see how many of the top contributors were responsible for having 80% of the accumulated points in Discogs. When looking at contributions of the top 1000 contributors 60% of the contributors accounted for about 80% of the points. The more data I got the more this moved towards 20% and it became clear very quickly that Discogs indeed seems to follow the 80/20 rule: when looking at the points of the top 36,000 contributors 80% of the points accumulated belong to the top 21.3% contributo...

In Discogs you can get points for every contribution. There are a few people who have made it a sport to score as many points as possible, the so called "rank hunters" which some people look down on for reasons that I do not understand (in case you want to be a rankhunter or you want to be a more effective one, please check out the "Unofficial Discogs rankhunting guide" to maximize your efforts). It works like this: adding a new release gets you three points and each edit (regardless how much you edit) gives you one point and there is some sort of leaderboard/ranking for all contributing users to Discogs . There are a few people that have an enormous amount of points and who seem to live for the site. The number one currently has over 356,000 points. When looking at the graphs they really reminded me of power laws , and the 80/20 rule because when looking at the first page of the contributors in Discogs and power law pictures there is a striking resemblance so...

Time to look into a few more aspects of the Discogs data model and where I think it could be improved. If you haven't read the first part I would suggest you do that first. This time I want to look at a few more: rights society, mastering and mould SID codes, and pressing plant. Rights Society Currently rights societies are stored in the "Barcodes and Other Identifiers" (BaOI) section which covers the whole release. The correct place for this information would actually be per track. There are some releases in Discogs (like some compilations) where it is indicated which rights society applies to which track. Mastering and Mould SID codes Like rights societies the mastering and mould SID codes are stored in the BaOI section that covers the whole release. The correct place for these identifiers would actually be a physical disc, as they frequently are different in releases with multiple discs per release. The solution that people use is to tag it in the free text fie...

After looking at many releases in Discogs and the edit history of various releases and how things have changed through time it is quite clear to see how the data model has changed. Some of these changes have turned out well, while others were, in retrospect, probably not the right change to make. I am not blaming the developers as I know from experience how difficult it is to get a data model right the first time and how hard it is to change: as soon as something is in use it is non-trivial to change especially with a database the size of Discogs. Still I want to go through a few examples where I think the current datamodel doesn't work. Most of my examples will focus on the fields from the "Barcodes and Other Identifiers" (BaOI) section. In this post I am looking at three of them: ISRC, SPARS codes and matrix/runout. ISRC Currently ISRC codes (International Standard Recording Code, see previous articles about it for more information: 1 , 2 , 3 ) are stored in the ...

I decided to look into character sets again, because the last time I only looked at a few instances of "homoglyphs" (characters that look like characters in other languages, but which are different). It is recommended to first read the first article about this. I looked into rights societies again. Last time I mostly focused on Greek characters, this time I looked at where Latin characters were used where Cyrillic characters should have been used. The rights society in Russia is called RAO which in Russian is spelled РАО, which looks a lot like PAO, but it is different (first one is using the Cyrillic alphabet, second one the Latin alphabet). They are in the data as follows: Distribution of wrong entries for Russian rights society РАО What is interesting is the peak for recent releases. I don't know whether this is because more Russian releases have been added recently, or if this error has mostly been corrected for earlier releases.

A challenge in many data sets is how to properly deal with internationalization and characters. Discogs is no exception. In quite a few character sets there are characters that look exactly the same as characters in other character sets, but they aren't. Take for example the following two instances of a Rights Society name that I found in Discogs: BIEM ΒΙΕΜ To the naked eye they might look the same (depending on the font the website is using), but they actually are not the same. The first one is all ASCII, but in the second one all the characters are from the Greek alphabet. From a data point of view these characters are very different! Characters that look a lot like other characters are called homoglyphs . I wrote about this before when looking at misspellings in Czechoslovak and Czech releases (which was essentially the same problem). There are many examples of homoglyphs in the Discogs database and it can get confusing. For example, let's look at entries for a Gree...

After having seen quite a few releases on Discogs and having dug through the data that Discogs makes available every month I am sometimes still finding things that make me wonder. In a previous post I already talked a bit about what I could do if the whole edit history would be made available, but I forgot one important thing: some people tend to put information that is relevant to the release in the submission notes. The submission notes are just metadata describing an edit . They should not be used to describe a release . Yet some people do exactly this. They do not realize that unless you are logged into Discogs and actively click on "edit page" you will not see the submission history and you will not see that particular information. It is effectively hidden for people who do not know this or who are not logged in. To find releases with that information it would be really useful to have the Discogs edit history. On the other hand, maybe it should also be a signal to...

Discogs is making its database data available under a CC0 license, but one important aspect of the data is lacking and that is the edit history. For me as a software engineer having the edit history available is very natural, as it allows me to dive into the history perhaps see where people went wrong, when certain errors were introduced, and so on. For Discogs this is only possible if I am logged into the website and go to the release page history where I am presented with a view of the edit history without being able to interact with it (queries, and so on), which I find frustrating. The edit history is a very rich place of information. For each release a lot of information is kept, such as: creation date/modifcation date origin (whether it was created fresh, or if "copy to draft" was used) merge information (if merged with another release) content of edits contributors and votes While some of this information is partially available through the Discogs API the ful...

One feeling that I have when I am looking through Discogs is "if only they would have done XYZ". I understand that there is always more to do than there is manpower available and what I perceive as a big problem might be very low priority (and vice versa), but that doesn't stop me from ranting about it. This is one of those posts. In Discogs it is expected that people upload images. The guidelines for images describe how this is being done and they make a lot of sense. For example, if a release (like a 7" single) has a sleeve and two sides the first image you should show is the front cover, then back cover, then the A-side label, then the B-side label. Of course, humans being humans it is completely random at times (although many times it is perfectly fine). Some things I have observed: the order can be quite random it is unclear what the images are off some people feel the need to upload images of their own copy people add images of another release or worse...