Tokens, Patterns and Lexemes
- A token is a pair consisting of a token name and an optional attribute value. The token name is an abstract symbol representing a king of lexical unit, e.g., a particular keyword or a sequence of input characters denoting an identifier.
- A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword.
- A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token.
Example:
Identify the lexeme and the respective token in the following C-statement.
printf(“Total = %d\n”, score);
Solution:
printf is lexeme corresponding to identifier score is lexeme corresponding to identifier
“Total = %d\n” is a lexeme corresponding to literal.
Dear Aspirants,
Your preparation for GATE, ESE, PSUs, and AE/JE is now smarter than ever — thanks to the MADE EASY YouTube channel.
This is not just a channel, but a complete strategy for success, where you get toppers strategies, PYQ–GTQ discussions, current affairs updates, and important job-related information, all delivered by the country’s best teachers and industry experts.
If you also want to stay one step ahead in the race to success, subscribe to MADE EASY on YouTube and stay connected with us on social media.
MADE EASY — where preparation happens with confidence.
MADE EASY is a well-organized institute, complete in all aspects, and provides quality guidance for both written and personality tests. MADE EASY has produced top-ranked students in ESE, GATE, and various public sector exams. The publishing team regularly writes exam-related blogs based on conversations with the faculty, helping students prepare effectively for their exams.
