The Battle for Wesnoth  1.15.1+dev
Public Member Functions | Private Types | Private Member Functions | Private Attributes | List of all members
tokenizer Class Reference

Abstract baseclass for the tokenizer. More...

#include <tokenizer.hpp>

Public Member Functions

 tokenizer (std::istream &in)
 
 ~tokenizer ()
 
const tokennext_token ()
 
const tokencurrent_token () const
 
const std::string & textdomain () const
 
const std::string & get_file () const
 
int get_start_line () const
 

Private Types

enum  { TOK_NONE = 0, TOK_SPACE = 1, TOK_NUMERIC = 2, TOK_ALPHA = 4 }
 

Private Member Functions

 tokenizer ()
 
void next_char ()
 
void next_char_fast ()
 
int peek_char ()
 
int char_type (unsigned c) const
 
bool is_space (int c) const
 
bool is_num (int c) const
 
bool is_alnum (int c) const
 
void skip_comment ()
 
bool skip_command (char const *cmd)
 Returns true if the next characters are the one from cmd followed by a space. More...
 

Private Attributes

int current_
 
int lineno_
 
int startlineno_
 
std::string textdomain_
 
std::string file_
 
token token_
 
buffered_istream in_
 
char char_types_ [128]
 

Detailed Description

Abstract baseclass for the tokenizer.

Definition at line 55 of file tokenizer.hpp.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum
private
Enumerator
TOK_NONE 
TOK_SPACE 
TOK_NUMERIC 
TOK_ALPHA 

Definition at line 132 of file tokenizer.hpp.

Constructor & Destructor Documentation

◆ tokenizer() [1/2]

tokenizer::tokenizer ( std::istream &  in)

◆ ~tokenizer()

tokenizer::~tokenizer ( )

Definition at line 47 of file tokenizer.cpp.

References in_, and buffered_istream::stream().

◆ tokenizer() [2/2]

tokenizer::tokenizer ( )
private

Member Function Documentation

◆ char_type()

int tokenizer::char_type ( unsigned  c) const
inlineprivate

Definition at line 140 of file tokenizer.hpp.

References c.

◆ current_token()

const token& tokenizer::current_token ( ) const
inline

Definition at line 63 of file tokenizer.hpp.

◆ get_file()

const std::string& tokenizer::get_file ( ) const
inline

Definition at line 80 of file tokenizer.hpp.

◆ get_start_line()

int tokenizer::get_start_line ( ) const
inline

Definition at line 85 of file tokenizer.hpp.

◆ is_alnum()

bool tokenizer::is_alnum ( int  c) const
inlineprivate

Definition at line 155 of file tokenizer.hpp.

Referenced by next_token().

◆ is_num()

bool tokenizer::is_num ( int  c) const
inlineprivate

Definition at line 150 of file tokenizer.hpp.

Referenced by skip_comment().

◆ is_space()

bool tokenizer::is_space ( int  c) const
inlineprivate

Definition at line 145 of file tokenizer.hpp.

Referenced by next_token(), skip_command(), and skip_comment().

◆ next_char()

void tokenizer::next_char ( )
inlineprivate

Definition at line 96 of file tokenizer.hpp.

Referenced by next_token().

◆ next_char_fast()

void tokenizer::next_char_fast ( )
inlineprivate

Definition at line 103 of file tokenizer.hpp.

Referenced by next_token(), skip_command(), skip_comment(), and tokenizer().

◆ next_token()

const token & tokenizer::next_token ( )

◆ peek_char()

int tokenizer::peek_char ( )
inlineprivate

Definition at line 127 of file tokenizer.hpp.

Referenced by next_token().

◆ skip_command()

bool tokenizer::skip_command ( char const *  cmd)
private

Returns true if the next characters are the one from cmd followed by a space.

Skips all the matching characters.

Definition at line 164 of file tokenizer.cpp.

References current_, is_space(), and next_char_fast().

Referenced by skip_comment().

◆ skip_comment()

void tokenizer::skip_comment ( )
private

Definition at line 176 of file tokenizer.cpp.

References current_, file_, is_num(), is_space(), lineno_, next_char_fast(), skip_command(), and textdomain_.

Referenced by next_token().

◆ textdomain()

const std::string& tokenizer::textdomain ( ) const
inline

Definition at line 75 of file tokenizer.hpp.

Member Data Documentation

◆ char_types_

char tokenizer::char_types_[128]
private

Definition at line 175 of file tokenizer.hpp.

Referenced by tokenizer().

◆ current_

int tokenizer::current_
private

Definition at line 92 of file tokenizer.hpp.

Referenced by next_token(), skip_command(), and skip_comment().

◆ file_

std::string tokenizer::file_
private

Definition at line 169 of file tokenizer.hpp.

Referenced by skip_comment().

◆ in_

buffered_istream tokenizer::in_
private

Definition at line 174 of file tokenizer.hpp.

Referenced by tokenizer(), and ~tokenizer().

◆ lineno_

int tokenizer::lineno_
private

Definition at line 93 of file tokenizer.hpp.

Referenced by next_token(), and skip_comment().

◆ startlineno_

int tokenizer::startlineno_
private

Definition at line 94 of file tokenizer.hpp.

Referenced by next_token().

◆ textdomain_

std::string tokenizer::textdomain_
private

Definition at line 168 of file tokenizer.hpp.

Referenced by skip_comment().

◆ token_

token tokenizer::token_
private

Definition at line 170 of file tokenizer.hpp.

Referenced by next_token().


The documentation for this class was generated from the following files: