utf8 Namespace Reference

Functions for converting Unicode wide-char strings to UTF-8 encoded strings, back and forth. More...

Classes
class	invalid_utf8_exception
	Thrown by operations encountering invalid UTF-8 data. More...

Typedefs
typedef ucs4::iterator_base< std::string_view, ucs4_convert_impl::convert_impl< char >::type >	iterator

Functions
static int	byte_size_from_utf8_first (const unsigned char ch)
std::string	lowercase (std::string_view s)
	Returns a lowercased version of the string. More...
std::size_t	index (std::string_view str, const std::size_t index)
	Codepoint index corresponding to the nth character in a UTF-8 string. More...
std::size_t	size (std::string_view str)
	Length in characters of a UTF-8 string. More...
std::string &	insert (std::string &str, const std::size_t pos, const std::string &insert)
	Insert a UTF-8 string at the specified position. More...
std::string &	erase (std::string &str, const std::size_t start, const std::size_t len=std::string::npos)
	Erases a portion of a UTF-8 string. More...
std::string &	truncate (std::string &str, const std::size_t size)
	Truncates a UTF-8 string to the specified number of characters. More...
void	truncate_as_ucs4 (std::string &str, const std::size_t size)
	Truncates a UTF-8 string to the specified number of characters. More...

Detailed Description

Functions for converting Unicode wide-char strings to UTF-8 encoded strings, back and forth.

Typedef Documentation

iterator

typedef ucs4::iterator_base<std::string_view, ucs4_convert_impl::convert_impl<char>::type> utf8::iterator

Definition at line 41 of file unicode.hpp.

Function Documentation

byte_size_from_utf8_first()

static int utf8::byte_size_from_utf8_first ( const unsigned char  ch )

static

Definition at line 35 of file unicode.cpp.

References count_leading_ones().

Referenced by index(), and size().

erase()

std::string & utf8::erase	(	std::string &	str,
		const std::size_t	start,
		const std::size_t	len = std::string::npos
	)

Erases a portion of a UTF-8 string.

Parameters

str	UTF-8 encoded string.
start	Start position.
len	Number of characters to erase.

Note

This implementation does not check for valid UTF-8. Don't use it for user input.

Definition at line 103 of file unicode.cpp.

References index(), size(), and editor::start().

Referenced by gui2::combobox::delete_selection(), gui2::multiline_text::delete_selection(), gui2::password_box::delete_selection(), gui2::text_box::delete_selection(), carryover::get_recruits(), campaignd::server::handle_delete(), ai::simulated_attack(), and truncate().

index()

std::size_t utf8::index	(	std::string_view	str,
		const std::size_t	index
	)

Codepoint index corresponding to the nth character in a UTF-8 string.

Returns

str.length() if there are less than index characters.

Definition at line 70 of file unicode.cpp.

References byte_size_from_utf8_first(), ERR_GENERAL, and i.

Referenced by pathfind::a_star_search(), unit::ability_active_impl(), display::action_pressed(), actions::actor_sighted(), gui2::tree_view_node::add_child(), config::add_child_at(), simple_wml::node::add_child_at(), config::add_child_at_total(), gui2::tree_view_node::add_child_impl(), gui2::tree_view::add_node(), gui2::listbox::add_row(), config::apply_diff(), simple_wml::node::apply_diff(), BOOST_AUTO_TEST_CASE(), gui2::dialogs::units_dialog::build_create_dialog(), gui2::dialogs::units_dialog::build_recall_dialog(), gui2::dialogs::units_dialog::build_recruit_dialog(), gui2::dialogs::units_dialog::build_unit_list_dialog(), team::calculate_enemies(), team::calculate_is_enemy(), editor::editor_controller::can_execute_command(), ai::cfun_ai_get_suitable_keep(), wesnothd::server::cleanup_game(), config::clear_diff_track(), persist_file_context::clear_var(), gui2::chatbox::close_window_button_callback(), gui2::text_box_base::copy_selection(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::create_item(), gui2::policy::minimum_selection::one_item::create_item(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::create_items(), gui2::dialogs::unit_attack::damage_calc_callback(), gui2::dialogs::game_load::delete_button_callback(), gui2::policy::minimum_selection::one_item::delete_item(), gui2::policy::minimum_selection::no_item::delete_item(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::delete_item(), gui2::policy::minimum_selection::one_item::deselect_item(), gui2::policy::minimum_selection::no_item::deselect_item(), events::map_command_handler< Worker >::dispatch(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::do_deselect_item(), editor::editor_controller::do_execute_command(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::do_select_item(), gui2::dialogs::mp_lobby::enter_game_by_index(), erase(), gui2::dialogs::game_load::execute(), gui2::dialogs::mp_join_game::fetch_game_config(), pathfind::find_routes(), ng::flg_manager::find_suitable_faction(), t_translation::ter_map::get(), editor::editor_controller::get_action_state(), tod_manager::get_area_by_index(), gui2::styled_widget::get_canvas(), variable_info_implementation::vi_policy_create::get_child_at(), variable_info_implementation::vi_policy_throw::get_child_at(), variable_info_implementation::vi_policy_const::get_child_at(), gui2::tree_view_node::get_child_at(), font::pango_text::get_column_line(), version_info::get_component(), tod_manager::get_current_area_time(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::get_item_shown(), font::pango_text::get_line(), gui2::text_box_base::get_line(), hotkey::command_executor::get_menu_image(), get_noise(), wb::side_actions::get_numbers(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::get_ordered_index(), gui2::implementation::placer_horizontal_list::get_origin(), gui2::implementation::placer_vertical_list::get_origin(), storyscreen::controller::get_part(), team::get_side_color_id(), team::get_side_color_range(), wesnothd::game::get_side_player(), font::pango_text::get_token(), gui2::dialogs::addon_manager::get_type_filter_visibility(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::impl_create_items(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::impl_draw_children(), schema_validation::inheritance_loop_error(), insert(), replay_recorder_base::insert_command(), ng::depcheck::manager::insert_element(), spirit_po::catalog< hashmap_type, pf_compiler >::insert_message(), font::pango_text::insert_text(), internal_get_unit(), game_lua_kernel::intf_get_sides(), intf_music_add(), replay_recorder_base::is_ancestor(), team::is_enemy(), ng::depcheck::manager::is_modification_active(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::is_selected(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::item(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::item_ordered(), team::knows_about_team(), editor::context_manager::load_mru_item(), luaW_check_mgfilter(), LuaW_checkcolor(), luaW_checkconfig(), luaW_checklocation(), luaW_checkterrainmap(), luaW_checktstring(), luaW_checkunit(), luaW_checkunit_ptr(), luaW_checkunit_ref(), luaW_checkvconfig(), luaW_is_mgfilter(), luaW_iscolor(), luaW_isterrainmap(), luaW_iststring(), luaW_isunit(), luaW_iswidget(), luaW_table_get_def(), luaW_table_set(), luaW_tableget(), luaW_to_locationset(), luaW_to_mgfilter(), luaW_toconfig(), luaW_tolocation(), luaW_toscalar(), luaW_tostring(), luaW_tostring_or_default(), luaW_toterrainmap(), luaW_totstring(), luaW_tounit(), luaW_tounit_ptr(), luaW_tounit_ref(), luaW_tovconfig(), wesnothd::make_add_diff(), wesnothd::make_change_diff(), wesnothd::make_delete_diff(), terrain_filter::match_internal(), display::menu_pressed(), gui2::dialogs::addon_manager::on_order_changed(), variable_info_implementation::replace_range_h::operator()(), variable_info_implementation::merge_range_h::operator()(), recall_list_manager::operator[](), cave_map_generator::cave_map_generator_job::place_chamber(), gui2::pane::place_children(), gui2::pane::place_or_set_origin_children(), ai::readonly_context_impl::power_projection(), gui2::dialogs::addon_manager::pre_show(), gui2::dialogs::mp_create_game::pre_show(), playmp_controller::process_network_change_controller_impl(), playmp_controller::process_network_side_drop_impl(), wesnothd::game::process_turn(), ai::push_move_map(), hotkey::command_executor::queue_command(), read_rules_vector(), metrics::record_sample(), editor::map_context::remove_area(), config::remove_child(), simple_wml::node::remove_child(), config::remove_children(), replay_recorder_base::remove_command(), replay::remove_command(), ai::ai_default_rca::candidate_action_evaluation_loop::remove_completed_cas(), wesnothd::server::remove_player(), desktop::remove_user_bookmark(), editor::map_context::select_area(), gui2::policy::maximum_selection::one_item::select_item(), gui2::policy::maximum_selection::many_items::select_item(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::select_item(), gui2::tab_container::select_tab(), editor::map_context::set_active_area(), gui2::dialogs::units_dialog::set_column(), version_info::set_component(), ng::create_engine::set_current_era_index(), ng::flg_manager::set_current_faction(), ng::flg_manager::set_current_gender(), ng::flg_manager::set_current_leader(), ng::create_engine::set_current_level(), gui2::dialogs::editor_resize_map::set_direction_icon(), editor::palette_manager::set_group(), editor::editor_palette< Item >::set_group(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::set_item_selected(), gui2::policy::minimum_selection::one_item::set_item_shown(), gui2::policy::minimum_selection::no_item::set_item_shown(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::set_item_shown(), gui2::pane::set_origin_children(), gui2::dialogs::simple_item_selector::set_selected_index(), gui2::dialogs::theme_list::set_selected_index(), editor::editor_palette< Item >::set_start_item(), editor::location_palette::set_start_item(), gui2::dialogs::units_dialog::set_tooltip_generator(), attack_type::special_active_impl(), editor::context_manager::switch_context(), tod_manager::times(), gui2::generator_base::toggle_item(), theme::action::tooltip(), ng::depcheck::manager::try_era_by_index(), ng::depcheck::manager::try_scenario_by_index(), unit_show_error(), gui2::dialogs::mp_create_game::update_games_list(), and font::pango_text::xy_to_index().

insert()

std::string & utf8::insert	(	std::string &	str,
		const std::size_t	pos,
		const std::string &	insert
	)

Insert a UTF-8 string at the specified position.

Definition at line 98 of file unicode.cpp.

References index().

Referenced by wfl::builtins::DEFINE_WFL_FUNCTION(), gui2::text_box_base::handle_editing(), gui2::dialogs::title_screen::init_callbacks(), pathfind::paths::dest_vect::insert(), gui2::password_box::insert_char(), font::pango_text::insert_text(), wesnothd::make_add_diff(), and wesnothd::make_change_diff().

lowercase()

std::string utf8::lowercase

(

std::string_view

s

)

Returns a lowercased version of the string.

Definition at line 50 of file unicode.cpp.

References ucs4::iterator_base< string_type, update_implementation >::end(), s, and ucs4::iterator_base< string_type, update_implementation >::substr().

Referenced by wesnothd::server::bans_handler(), BOOST_AUTO_TEST_CASE(), wesnothd::server::dul_handler(), campaignd::blacklist::is_in_globlist(), wesnothd::server::is_login_allowed(), filesystem::looks_like_pbl(), wesnothd::ban_manager::parse_time(), wesnothd::server::process_command(), wesnothd::server::searchlog_handler(), wesnothd::server::status_handler(), gui2::dialogs::chat_log::model::stream_log(), and campaignd::server::validate_addon().

size()

std::size_t utf8::size

(

std::string_view

str

)

Length in characters of a UTF-8 string.

Definition at line 85 of file unicode.cpp.

References byte_size_from_utf8_first(), ERR_GENERAL, and i.

Referenced by _crypt_output_magic(), wb::action::action(), gui2::rich_label::add_attribute(), font::pango_text::add_attribute_size(), gui2::implementation::placer_horizontal_list::add_item(), gui2::implementation::placer_vertical_list::add_item(), gui2::rich_label::add_link(), gui2::dialogs::mp_staging::add_side_node(), wb::attack::attack(), battle_context::battle_context(), BF_crypt(), BF_decode(), BF_encode(), BOOST_AUTO_TEST_CASE(), gui2::implementation::builder_drawing::build(), cave_map_generator::cave_map_generator_job::build_chamber(), gui2::grid::calculate_best_size(), gui2::matrix::calculate_best_size(), gui2::size_lock::calculate_best_size(), gui2::spacer::calculate_best_size(), team::calculate_enemies(), font::pango_text::calculate_size(), editor::editor_controller::can_execute_command(), ai::cfun_ai_get_recruitment_pattern(), battle_context::choose_attacker_weapon(), battle_context::choose_defender_weapon(), shroud_map::clear(), gui2::grid_implementation::column_request_reduce_width(), gui2::scrollbar_container::content_resize_request(), server_base::coro_receive_doc(), server_base::coro_send_doc(), coro_send_file_userspace(), gui2::dialogs::chat_log::model::count_of_pages(), play_controller::current_team(), cut_surface(), wfl::builtins::DEFINE_WFL_FUNCTION(), wfl::gamestate::DEFINE_WFL_FUNCTION(), editor::editor_controller::do_execute_command(), do_replay_handle(), wb::draw_numbers(), wesnothd::server::dummy_player_updates(), utils::ellipsis_truncate(), erase(), combatant::fight(), filesystem::file_size(), events::mouse_handler::fill_weapon_choices(), prefs::font_scaled(), footsteps_images(), wfl::formula_function_expression::formula_function_expression(), game_launcher::game_launcher(), gui2::styled_widget::get_best_text_size(), wfl::function_symbol_table::get_builtins(), gui2::text_box_base::get_composition_length(), gui2::tree_view_node::get_current_size(), font::floating_label::get_draw_size(), ai::default_recruitment::recruitment::get_estimated_village_gain(), filesystem::get_exe_path(), filesystem::get_files_in_dir(), font::get_floating_label_rect(), gui2::tree_view_node::get_folded_size(), font::get_max_height(), gui2::rich_label::get_parsed_text(), display::get_submerge_data(), prefs::get_system_username(), gui2::tree_view_node::get_unfolded_size(), ai::formula_ai::get_value(), gui2::text_box_base::handle_editing(), gui2::text_box_base::handle_key_right_arrow(), display_context::has_team(), game_events::builtin_conditions::have_unit(), filesystem::ifs_read(), display::init_flags(), gui2::text_box_base::insert_char(), gui2::password_box::insert_char(), font::pango_text::insert_text(), game_lua_kernel::intf_create_side(), intf_default_generate_height_map(), intf_describe_plugins(), game_lua_kernel::intf_find_cost_map(), game_lua_kernel::intf_find_path(), game_lua_kernel::intf_find_reach(), game_lua_kernel::intf_get_side(), game_lua_kernel::intf_is_enemy(), game_lua_kernel::intf_match_side(), game_lua_kernel::intf_put_recall_unit(), lua_fileops::intf_read_file(), game_lua_kernel::intf_set_floating_label(), game_lua_kernel::intf_set_side_id(), game_lua_kernel::intf_toggle_fog(), editor::editor_display::layout(), gui2::window::layout(), gui2::grid::layout(), gui2::window_implementation::layout(), gui2::window::layout_linked_widgets(), lua_fileops::lua_filestream::lua_read_data(), hotkey::hotkey_keyboard::matches_helper(), filesystem::ofs_write(), image::scale_modification::operator()(), font::p_font::p_font(), font::pango_draw_text(), font::pango_render_text(), php_crypt_blowfish_rn(), gui2::policy::placement::table::place(), gui2::widget::place(), gui2::combobox::place(), gui2::container_base::place(), gui2::policy::placement::horizontal_list::place(), gui2::policy::placement::vertical_list::place(), gui2::policy::placement::independent::place(), gui2::generator< minimum_selection, maximum_selection, my_placement, select_action >::place(), gui2::grid::place(), gui2::listbox::place(), gui2::matrix::place(), gui2::multiline_text::place(), gui2::pane::place(), gui2::scroll_text::place(), gui2::scrollbar_base::place(), gui2::scrollbar_container::place(), gui2::size_lock::place(), gui2::slider_base::place(), gui2::styled_widget::place(), gui2::text_box::place(), gui2::tree_view_node::place(), gui2::viewport::place(), shroud_map::place(), gui2::grid::child::place(), gui2::text_box_base::plain_text(), playsingle_controller::play_scenario_main_loop(), events::raise_resize_event(), ucs4_convert_impl::utf8_impl::read(), gui2::container_base::reduce_height(), gui2::grid::reduce_height(), gui2::container_base::reduce_width(), gui2::grid::reduce_width(), font::relative_size(), gui2::grid::request_placement(), gui2::container_base::request_reduce_height(), gui2::grid::request_reduce_height(), gui2::scrollbar_container::request_reduce_height(), gui2::styled_widget::request_reduce_height(), gui2::container_base::request_reduce_width(), gui2::grid::request_reduce_width(), gui2::scrollbar_container::request_reduce_width(), gui2::styled_widget::request_reduce_width(), gui2::listbox::resize_content(), gui2::tree_view::resize_content(), variable_info_implementation::resolve_negative_value(), editor::map_fragment::rotate_60_ccw(), editor::map_fragment::rotate_60_cw(), gui2::grid_implementation::row_request_reduce_height(), prefs::save_sound_buffer_size(), wb::manager::save_temp_move(), server_base::send_doc_queued(), wb::manager::send_network_data(), gui2::scrollbar_container::set_content_size(), gui2::listbox::set_content_size(), gui2::unit_preview_pane::set_display_data(), editor::editor_display::set_help_string(), gui2::dialogs::file_dialog::set_input_text(), gui2::widget::set_layout_size(), display::set_playing_team_index(), gui2::canvas::set_size(), gui2::widget::set_size(), editor::editor_display::set_status(), editor::mouse_action::set_terrain_mouse_overlay(), gui2::password_box::set_value(), display::set_viewing_team_index(), size_display_string(), markup::span_size(), draw::tiled_highres(), video::to_output(), wesnothd::game::transfer_side_control(), truncate(), truncate_as_ucs4(), unit_weapons(), gui2::dialogs::game_cache_options::update_cache_size_display(), gui2::multiline_text::update_canvas(), gui2::dialogs::mp_lobby::update_gamelist(), gui2::dialogs::mp_lobby::update_gamelist_diff(), gui2::listbox::update_layout(), display::update_render_textures(), gui2::dialogs::chat_log::controller::update_view_from_model(), validate_side(), shroud_map::value(), game_events::WML_HANDLER_FUNCTION(), and network::write_callback().

truncate()

std::string & utf8::truncate	(	std::string &	str,
		const std::size_t	size
	)

Truncates a UTF-8 string to the specified number of characters.

Parameters

str	UTF-8 encoded string.
size	Size to truncate to.

Note

This implementation does not check for valid UTF-8. Don't use it for user input.

Definition at line 116 of file unicode.cpp.

References erase(), and size().

Referenced by BOOST_AUTO_TEST_CASE(), utils::ellipsis_truncate(), and font::pango_text::set_maximum_length().

truncate_as_ucs4()

void utf8::truncate_as_ucs4	(	std::string &	str,
		const std::size_t	size
	)

Truncates a UTF-8 string to the specified number of characters.

If the string has more than size UTF-8 characters it will be truncated to this size.

The output is guaranteed to be valid UTF-8.

Parameters

[in,out]	str	[in] String encoded in UTF-8. [out] String encoded UTF-8 that contains at most size codepoints.
	size	The size to truncate to.

Definition at line 121 of file unicode.cpp.

References size().

Referenced by wesnothd::chat_message::truncate_message().