contains the current text being parsed as well as the token_type of what's being parsed. More...

#include <tokenizer.hpp>

Public Types
enum	token_type { STRING , QSTRING , UNTERMINATED_QSTRING , MISC , NEWLINE = '\n' , EQUALS = '=' , COMMA = ',' , PLUS = '+' , SLASH = '/' , OPEN_BRACKET = '[' , CLOSE_BRACKET = ']' , UNDERSCORE = '_' , END = 256 }
	used for a token's type field More...

enum	source_chars { POUND = '#' , LEFT_ANGLE_BRACKET = '<' , RIGHT_ANGLE_BRACKET = '>' , DOUBLE_QUOTE = '"' , DOLLAR = '$' }
	not used for a token's type field More...

Public Member Functions
	token ()

Public Attributes
token_type	type

std::string	value
	the token's value, can be either a single character or multiple characters More...

Detailed Description

contains the current text being parsed as well as the token_type of what's being parsed.

multi-character token types will have a value that's a string with zero or more characters in it. single character token types are a single character with special meaning for a config

Definition at line 40 of file tokenizer.hpp.

Member Enumeration Documentation

◆ source_chars

enum token::source_chars

not used for a token's type field

Enumerator
POUND
LEFT_ANGLE_BRACKET
RIGHT_ANGLE_BRACKET
DOUBLE_QUOTE
DOLLAR

Definition at line 79 of file tokenizer.hpp.

◆ token_type

enum token::token_type

used for a token's type field

Enumerator
STRING	unquoted text
QSTRING	quoted string, contained within double quotes or by less than/greater than symbols
UNTERMINATED_QSTRING	reached end of file without finding the closing character for a QSTRING
MISC	any characters that don't have special meaning
NEWLINE
EQUALS
COMMA
PLUS
SLASH
OPEN_BRACKET
CLOSE_BRACKET
UNDERSCORE
END	set when EOF is returned by the input stream

Definition at line 50 of file tokenizer.hpp.

Constructor & Destructor Documentation

◆ token()

token::token ( )

inline

Definition at line 42 of file tokenizer.hpp.

Member Data Documentation

◆ type

token_type token::type

Definition at line 88 of file tokenizer.hpp.

Referenced by preprocessor_data::get_chunk(), and tokenizer::next_token().

◆ value

std::string token::value

the token's value, can be either a single character or multiple characters

Definition at line 90 of file tokenizer.hpp.

Referenced by tokenizer::next_token().

The documentation for this struct was generated from the following file:

serialization/tokenizer.hpp

Public Types

Public Member Functions

Public Attributes