token Struct Reference

#include <tokenizer.hpp>

Public Types
enum	token_type {   STRING , QSTRING , UNTERMINATED_QSTRING , MISC ,   LF = '\n' , EQUALS = '=' , COMMA = ',' , PLUS = '+' ,   SLASH = '/' , OPEN_BRACKET = '[' , CLOSE_BRACKET = ']' , UNDERSCORE = '_' ,   END }

Public Member Functions
	token ()

Public Attributes
token_type	type
std::string	value

Detailed Description

Definition at line 26 of file tokenizer.hpp.

Member Enumeration Documentation

token_type

enum token::token_type

Enumerator
STRING
QSTRING
UNTERMINATED_QSTRING
MISC
LF
EQUALS
COMMA
PLUS
SLASH
OPEN_BRACKET
CLOSE_BRACKET
UNDERSCORE
END

Definition at line 33 of file tokenizer.hpp.

Constructor & Destructor Documentation

token()

token::token ( )

inline

Definition at line 28 of file tokenizer.hpp.

Member Data Documentation

type

token_type token::type

Definition at line 51 of file tokenizer.hpp.

Referenced by preprocessor_data::get_chunk(), and tokenizer::next_token().

value

std::string token::value

Definition at line 52 of file tokenizer.hpp.

Referenced by tokenizer::next_token().

---

The documentation for this struct was generated from the following file:

• serialization/tokenizer.hpp