Proposed resolution (October, 2017):

1. Change 5.9 [lex.digraph] paragraph 2 as follows

In all respects of the language except in an attribute-token (9.12.1 [dcl.attr.grammar]), each alternative token behaves the same, respectively, as its primary token, except for its spelling.¹⁸ The set of alternative tokens...

2. Change the grammar in 5.10 [lex.token] as follows:

token:
identifier
keyword
literal-token
operator
punctuator


literal-token:
integer-literal
floating-literal
character-literal
string-literal
user-defined-literal


symbol: one of

{    }    [    ]    (    )    ;    :    ...  ?    ::   .    .*
+    -    *    /    %    ^    &    |    ~    !    =    <    >
+=   -=   *=   /=   %=   ^=   &=   |=   <<   >>   <<=  >>=
==   !=   <=   >=   &&   ||   ++   --   ,    ->*  ->

3. Change 5.10 [lex.token] paragraph 1 as follows:

There are five four kinds of tokens: identifiers, keywords, literals,¹⁹ operators, and other separators and symbols. Blanks, horizontal and vertical tabs, newlines, formfeeds, and comments (collectively, “white space”), as described below, are ignored except as they serve to separate tokens. [Note: Some white space is required to separate otherwise adjacent identifiers, keywords, numeric literals, and alternative tokens containing alphabetic characters. —end note] Each preprocessing-token resulting from translation phase 6 is converted into the corresponding token as follows:

• If the preprocessing-token is an identifier or is one of the preprocessing-op-or-punc tokens new or delete, the resulting token is a keyword if it is listed in Table 5, and otherwise is an identifier.

• Otherwise, if the preprocessing-token is a pp-number with the lexical form of an integer-literal or floating-literal, or is a character-literal or string-literal, the resulting token is of the corresponding form.

• Otherwise, if the preprocessing-token is a pp-number with the lexical form of a user-defined-integer-literal or user-defined-floating-literal or is a user-defined-character-literal or user-defined-string-literal, the resulting token is a user-defined-literal.

• Otherwise, if the preprocessing-token is a preprocessing-op-or-punc, and there is a corresponding symbol (after converting alternative token representations to their primary tokens), the resulting token is that symbol.

• Otherwise, the program is ill-formed.

[Note: Within an attribute-token (9.12.1 [dcl.attr.grammar]), a token formed from a preprocessing-token that satisfies the syntactic requirements of an identifier is considered to be an identifier with the spelling of the preprocessing-token. —end note]

4. Delete the final sentence of 5.8 [lex.operators] paragraph 1.

Each preprocessing-op-or-punc is converted to a single token in translation phase 7 (5.2 [lex.phases]).

The syntactic nonterminal punctuator appears in the grammar for token in 5.10 [lex.token], but it is nowhere defined. It should be merged with operator and given an appropriate list of tokens as a definition for the merged term.