tokenizer.hpp File Reference

#include "buffered_istream.hpp"
#include <array>
#include <istream>
#include <string>

Include dependency graph for tokenizer.hpp:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
struct	token
	contains the current text being parsed as well as the token_type of what's being parsed. More...
class	tokenizer
	class responsible for parsing the provided text into tokens and tracking information about the current token. More...

Variables
constexpr unsigned char	INLINED_PREPROCESS_DIRECTIVE_CHAR = 254
constexpr int	END_STANDARD_ASCII = 128

Variable Documentation

END_STANDARD_ASCII

constexpr int END_STANDARD_ASCII = 128

constexpr

Definition at line 33 of file tokenizer.hpp.

Referenced by tokenizer::char_type(), and tokenizer::tokenizer().

INLINED_PREPROCESS_DIRECTIVE_CHAR

constexpr unsigned char INLINED_PREPROCESS_DIRECTIVE_CHAR = 254

constexpr

Definition at line 29 of file tokenizer.hpp.

Referenced by preprocessor_data::get_chunk(), tokenizer::next_token(), preprocessor_data::preprocessor_data(), preprocessor_data::push_token(), preprocessor_data::put(), and preprocessor_streambuf::restore_old_preprocessor().