A look inside DMD

Tokenization

The first thing DMD does with D source code is split it up into 'tokens', which are the individual words, strings, numbers and punctuation. This is done by class Lexer, which creates instances of struct Token, which is a tagged union identified by a member in enum TOK : ubyte. Every D keyword has their own token type. Unfortunately, the compiler is in an awkward position where it needs to refer to its own reserved keywords a lot, so the enum names always have to be (slightly) different than the source code names. Often an underscore is added (pure becomes pure_), other times a different word is used (int becomes int32).

All other names that appear in source code (variables, functions, types, attributes, ...) create an 'identifier'. There isn't a hard line between keywords and identifiers though, since the compiler does have a long list of recognized idenfitiers. The main difference is that keywords often play an important role in letting the parser recognize constructs, like how if starts an if-statement and struct begins a struct declaration.

Interactive demo:

Some other observations:

• Number and string literals store their integer/string payload.
• There are different types of number literals: int32, int64, uns32 etc.
• The minus for negative numbers is a separate token: -3 is not parsed as int32literal(-3), but as min int32literal(3)
• Different kinds of string literal all create the same string_ token
• Builtin attributes with an @ are 2 tokens. So while nobody does it, you are allowed to write arbitrary white space and comments inbetween, like @ /*hello*/ nogc.
• Similarly, extern(C++) consists of separate identifier(C) and plusPlus tokens.

Tokenization is simple and stable. Personally, I rarely look at it. Most of the interesting stuff happens in the semantic analysis phase. Before we arrive there however, we first visit the next part: parsing

Parsing

Groups of tokens form 'constructs', like a class declaration, for loop, or expression. It's the parser's job to recognize these constructs and error on malformed syntax, like mismatched braces or missing semicolons. Type checking happens afterwards in the semantic phase.

If you ever wonder whether something is a parse error or a semantic error, try compiling the code inside a version (none) {} block. Alternatively, put it in a template that you don't instantiate. That way, the code doesn't get analyzed, but it is still required to parse.

Ambiguity

D tries to have a context free grammar. This means that theoretically, the parser can be implemented using functions like parseExp(), parseStatement(), parseDeclarations(), that turn the Lexer's token stream into Abstract Syntax Tree (AST) nodes, without any extra parameters or global variables to decide how to do that.

Contrast this to C, which infamously requires keeping a symbol table while parsing. Consider how the tokenizer ignores whitespace, so size_t* i; and x * y; both give the parser identifier mult identifier semicolon as input; It has to decide whether to create a declaration or expression based on what identifier refers to.

X *Y;  // If X came from `typedef int X`, declare Y as a pointer to X
X * Y; // If X and Y are variables, create a multiplication expression

Since D has forward declarations and parses context-free, how does it handle this case? Well, it arguably cheats a little. When X * Y can be parsed as a declaration, that always has priority over multiplication. This works because if it were a multiplication, it would be discarded, which is an error in D. (Well technically you can have structs with operator overloading and write X * Y for side-effects, but that's such bad style that I haven't ever seen anyone raise a complaint about that)

But there are also cases where the parser is too eager to parse a type, such as: foo!(X[Y]). That template parameter is assumed to be a static array of type X with dimension Y, but it could also be passing a compile time value computed by indexing X with Y. This is later handled in semantic analysis: if it resolves the TypeIdentifier to an expression, it rewrites the TypeSArray to an ArrayExp. Conversely, if in T[3].init the IdentifierExp resolves to a type, the ArrayExp gets rewritten to TypeSArray. Cheeky!

Interactive demo:

Bonus demos (selectable from a dropdown):