The lexical description

The Metal compiler creates a file, called Exp.tokens.x which provides the lexical description of Exp tokens. This file is used to construct a Lex scanner. It contains a list of regular expressions followed by actions. Usually, each action returns a token to the parser (via the return command). Returned tokens must bear the same names as the lexical variables that appear in atom production rules (in our case, %ID and %INTEGER).

The Exp.token.x file is merely a skeleton; it lacks the regular expressions that concern Exp. Thus, in the generated Exp.tokens.x file we find:


   <sentences>ID                return(ID);\
   <sentences>INTEGER           return(INTEGER);\

We replace the token variable names by the desired regular expressions:


   <sentences>[A-Z][A-Z0-9_]*   return(ID);\
   <sentences>[0-9][0-9]*       return(INTEGER);\

The above regular expressions mean that Exp identifiers must begin with a capital letter followed by any combination and number of capital letters, digits, and underscores. Integers must be at least one digit long. Note that we handle the minus sign that appears before negative numbers in the uminus concrete syntax definition.

N.B. Make sure to save the modified tokens file as Exp.tokens.


Tutorial