A challenge in many data sets is how to properly deal with internationalization and characters. Discogs is no exception. In quite a few character sets there are characters that look exactly the same as characters in other character sets, but they aren't. Take for example the following two instances of a Rights Society name that I found in Discogs:
There are many examples of homoglyphs in the Discogs database and it can get confusing. For example, let's look at entries for a Greek rights society I found when researching errors in the Rights Society field, where we can find that three character sets are mixed:
To make it even more interesting I also found 'ΑΕPI' in the data (first two characters Greek, last two ASCII) and 'AΕPI' (all but the second character Greek). For this particular rights society the correct way should either be all Greek, or all ASCII (depending on what is on the release). it should be possible to enforce this during data entry. I did not find entries where people had mixed Greek, Cyrillic and ASCII, or Greek and Cyrillic.
Discovering these errors is not easy to do using the Discogs search interface or by just going through release pages in Discogs. You actually can find the errors if you know the exact spelling mistakes that people have made. but by just browsing you would not spot them as they are not very common. The only way to discover this without spending endless amounts of time is automation.
For the rights societies I only tried to find the examples mentioned above, but I looked at not just the fields named Rights Society, but also the ones that should be. I found around 750 and they are distributed across the data as follows:
The difficulty fixing these errors is that most people probably will not appreciate it (seeing it as a frivolous fix) or even understand what the problem is even about ("it looks the same"). My best guess for what a solution could be is to hardcode as many of these values as possible and provide them as a drop down box (possibly with a free text field for any markup differences or a description).
My gut feeling tells me that I have barely begun to scratch the surface of this particular topic. I am already aware of some errors in the matrix/runout fields for old Soviet records, which I will look at in a future blogpost.
- BIEM
- ΒΙΕΜ
There are many examples of homoglyphs in the Discogs database and it can get confusing. For example, let's look at entries for a Greek rights society I found when researching errors in the Rights Society field, where we can find that three character sets are mixed:
- ΑΕΠΙ - all Greek
- AEΠΙ - first two characters ASCII, last two Greek
- AEΠI - all but the third character ASCII, third character Greek
- AEПI - all but the third character ASCII, third character Cyrillic
- AΕΠΙ - first character ASCII, rest Greek
- AΕΠI - first and last character ASCII, rest Greek
- АЕПI - last character ASCII, rest Cyrillic
- ΑΕΠΊ - all Greek
To make it even more interesting I also found 'ΑΕPI' in the data (first two characters Greek, last two ASCII) and 'AΕPI' (all but the second character Greek). For this particular rights society the correct way should either be all Greek, or all ASCII (depending on what is on the release). it should be possible to enforce this during data entry. I did not find entries where people had mixed Greek, Cyrillic and ASCII, or Greek and Cyrillic.
Discovering these errors is not easy to do using the Discogs search interface or by just going through release pages in Discogs. You actually can find the errors if you know the exact spelling mistakes that people have made. but by just browsing you would not spot them as they are not very common. The only way to discover this without spending endless amounts of time is automation.
Finding homoglyphs in the Discogs data
When looking at ASCII there are two character sets with which there will be the most confusion: Greek and Cyrillic, especially when using capitals. I tried to detect errors in several fields of the "Barcodes and Other Identifiers" section. A generic approach (seeing if people mixed ASCII and non-ASCII characters) turned out to be problematic with many false positives, as there were many different dots, spaces, slashes and pipes, from quite a few character sets, that obscured the real issue. I did find one instance of where a SPARS Code field had mixed Cyrillic and ASCII characters. As there are many instances where SPARS codes need to be fixed I am expecting that the real number will be a bit (but not much) higher.For the rights societies I only tried to find the examples mentioned above, but I looked at not just the fields named Rights Society, but also the ones that should be. I found around 750 and they are distributed across the data as follows:
 |
| Rights Society with values containing characters from a wrong character set |
The difficulty fixing these errors is that most people probably will not appreciate it (seeing it as a frivolous fix) or even understand what the problem is even about ("it looks the same"). My best guess for what a solution could be is to hardcode as many of these values as possible and provide them as a drop down box (possibly with a free text field for any markup differences or a description).
Other weird characters
There were a few other weird characters that I found just looking at the data. As already said there are many different dots, slashes, pipes and spaces but there are also a few very obscure characters like the "left-to-right mark". My guess is that this might have been entered by someone normally using "right-to-left". I have no idea how often this character actually is present in the data and I will leave that for another time.My gut feeling tells me that I have barely begun to scratch the surface of this particular topic. I am already aware of some errors in the matrix/runout fields for old Soviet records, which I will look at in a future blogpost.
Comments
Post a Comment