Unicode Gatekeepers
S. Gilles
2017-06-01
The problem
There has been a lot[0][1] of thought put into security implications
of unicode. A popular concern is that harmless string A might be
confused with evil string B by a human observer, who might then
be tricked into giving their login credentials to an attacker or
running code signed by an attacker.
That's not the part I'm interested in right now, however. There's
another concern: a system which accepts unicode allows spammers
a far wider playing field. It might not be a security risk for a
phishing attempt to include a link to paypa🄻.com, but it's annoying
to receive an email offering “DISCOUNT GENERIC 🄰 🅂 🄿 🄸 🅁 🄸 🄽 ”
because your spam filter was only trained on ASCII. In the language
of UTS #39, this is “fooling mechanical gatekeepers”.
UTS #39 is designed to handle the first concern. It includes (or
references, or describes, whatever) a file[2] named confusables.txt,
which defines a equivalences classes of unicode glyphs as the
consortium feels relevant. There are some other bits like multi-glyph
decomposition, but those aren't critical.
The problem is that there is no equivalent of confusables.txt for
gatekeepers. But suppose we really want one. Suppose we're building
software for a mailing list, or a BBS, or something, and we want
to give administrators the ability to detect unicode-obfuscated
spam by providing their own blacklist of keywords. How hard would
that be?
Initial reaction: ”Don't do that”
One obvious and completely foolproof method is to not support all
of unicode. Messages must be in ASCII, or Latin 1, or some other
small range.
The problem with this is that people tend to expect unicode support
these days. Even Slashcode was updated[3] to handle UTF-8 in 2014,
after all. For the sake of argument, let's assume that our biggest
customer has a habit of learning new languages and will be upset
if the software has to be reconfigured for each new alphabet.
Best Attempt
A second approach might be to map everything via confusables
before applying our checking. We might perform something along
the lines of (untested)
typedef struct {
Rune from;
Rune *to;
} replacement_t;
replacement_t reps[] = { ... };
#define NUM_REPLACEMENTS ((sizeof reps)/(sizeof reps[0]))
int rcomp(void *a, void *b)
{
Rune *k = a;
replacement_t *rep = b;
if (*k < rep->from) {
return -1;
} else if (*k > rep->from) {
return 1;
}
return 0;
}
int check_spam(const Rune *message, const Rune *badword)
{
Rune *normalized_message = 0;
Rune *normalized_badword = 0;
Rune *p = 0;
replacement_t *v = { 0 };
int ret = 0;
for (p = message; *p; p++) {
if ((v = bsearch(p, reps, NUM_REPLACEMENTS,
sizeof *reps, rcomp))) {
append_to_string(&normalized_message, v->to);
}
}
for (p = badword; *p; p++) {
if ((v = bsearch(p, reps, NUM_REPLACEMENTS,
sizeof *reps, rcomp))) {
append_to_string(&normalized_badword, v->to);
}
}
if (rune_strstr(normalized_message, normalized_badword)) {
ret = -1;
}
free(normalized_message);
free(normalized_badword);
return ret;
}
Aside from knowing the ‘...’ of reps, this isn't impossible, and
is even readable. If the algorithm flows this way, we can use sed
or awk to turn confusables.txt into the contents of ‘...’.
Issues
A few problems show up immediately.
o Extraneous characters. U+0224 LATIN CAPITAL LETTER Z WITH HOOK
should probably be replaced with a simple Z, but it is listed
as confusable with the sequence U+005A, U+0326 (a Z followed
by COMBINING COMMA BELOW).
o Overzealous replacing. “m” is confusable with “rn”, so it's
impossible to distinguish “yarn” and “yam”, in case we need
to censor vegetables or something. This could be a problem.
o Missing replacements. Codepoints like U+1F130 SQUARED LATIN
CAPITAL LETTER A should probably map to good old U+0041 LATIN
CAPITAL LETTER A. However, since the two glyps are visually
quite distinct, they are not considered confusable.
The last part is the most concerning. As newer releases of Unicode
offer more characters, confusables.txt may be updated, but since
there does not exist a “gatekeeper-confusables.txt”.
Finally
The solution I'm using is the following (generally) algorithm for
filling in the ‘...’ above. Depending on the real-world testing
it gets, it may need tweaking.
Parse confusables.txt into a map T: codepoint -> string.
Parse UnicodeData.txt[4], and
For every codepoint C
If the general category is M, C, P, or Sk
Make T delete that codepoint, by T(C) := ''
End
T(U+0020) := ' ' (it was mistakenly deleted above)
If there's no mapping
If UnicodeData.txt indicates a decomposition D1, D2, ...
T(C) = D1, D2, ...
End
End
End
(Now T deletes the ‘extra’ characters.)
Delete mappings for '0', '1', 'I', 'm', 'w', '|'.
Add mappings
T(U+0460) := 'w'
T(U+FF29) := 'I'
T(U+1F170) := 'A' (and so on through 'Z')
T(U+00A9) := 'C'
T(U+00AE) := 'R'
T(U+00DF) := 'B'
T(U+01AB) := 't'
T(U+0272) := 'n'
T(U+0274) := 'N'
T(U+0291) := 'z'
T(U+0298) := 'O'
T(U+029F) := 'L'
T(U+02B3) := 'r'
T(U+0629) := 'o'
T(U+0644) := 'J'
T(U+1472) := 'b'
T(U+1473) := 'b'
T(U+1D07) := 'E'
T(U+2117) := 'P'
T(U+2365) := 'O'
T(U+A793) := 'e'
T(U+2776) := '(1)' (and so on through '(10)')
T(U+2780) := '(1)' (and so on through '(10)')
T(U+278A) := '(1)' (and so on through '(10)')
Until T is fixed
T := T ∘ T
Output T in the form needed for rune transformation.
It would be nice for the Unicode Consortium to address this more
concretely.
[0] http://www.unicode.org/reports/tr36
[1] http://www.unicode.org/reports/tr39
[2] http://www.unicode.org/Public/security/latest/confusables.txt
[3] https://raw.githubusercontent.com/SoylentNews/rehash/fe00cc16d95c4b8bef921f9757a6af29712c7501/README.utf8
[4] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
