Unicode and Foreign Language Support
in Lexbe Litigation Support Software
Unicode support is important in litigation document management
as document repositories may include documents rendered with non-English
characters. These are typically produced in a version of Unicode, and a
litigation database that does not support unicode may not be able to display,
index or search foreign words. This document briefly describes Unicode in a litigation support context and
describes specifically foreign-language support through Unicode in the Lexbe
Unicode is a specification that allows text in any language to be
encoded in a consistent way. Detailed information on the Unicode
specification is maintained by the
Consortium. Computers initially used an encoding scheme called
Ascii to represent letters, but Ascii
is English-centric and does not allow sufficient characters to represent many
non-English characters. To remedy this deficiency, Unicode was developed.
Unicode can represent tens of thousands of distinct characters. Multiple versions of
Unicode are in use today, including UTF-8, UTF-16 and UTF-32. The UTF-8 and
UTF-16 versions are the most widely
Languages based on Latin-based alphabets can be viewed and created in nearly any
computer application without the need for any additional fonts. However, to type
or display foreign languages that use non-Latin or certain extended Latin-based
alphabets, the user may need to first download and install additional foreign
language fonts on his local computer. You can tell if you need to download a
font if characters appear as small rectangles. Microsoft Office contains a
useful "Arial Unicode MS" font with coverage of nearly every character in every
language included in the Unicode standard.
Unicode Support in Lexbe
Lexbe is primarily an English-language tool, but it does partially support
Unicode as described below:
Lexbe supports the 8-bit (UTF-8) and 16-bit (UCS-16)
encodings of Unicode, but not UCS-32.
Lexbe Unicode support means that it can index and
search documents containing Unicode-encoded data. Lexbe can also display
much Unicode in the Lexbe document browser, subject to client system
Lexbe can automatically recognize Unicode data in
Microsoft Word, Excel and PowerPoint files.
An HTML or XML file can include Unicode data if the
HTML file uses the UTF-8 encoding. Lexbe can index and search Unicode
data in UTF-8 encoded HTML files and can also recognize many other HTML
WordPerfect files use the WordPerfect Character Set
to express non-English text. Lexbe Online converts WordPerfect Character Set
data to Unicode for indexing, so non-English text in WordPerfect files
Lexbe can index and search Unicode characters in
some, but not all, PDF files, depending on how the PDF file was created.
Lexbe's concept-search functionality is supported for the
English language only.
Text in Chinese, Japanese, and Korean can be stored
in, or converted to, Unicode, so Lexbe Online can search for words in these
languages just as it can search for words in other languages. However,
while Lexbe can search for literal word matches (or wildcard or fuzzy
matches), there are some limitations on the support in Lexbe Online for
Chinese, Japanese, and Korean text, described below.
Some documents store text in a way that does not
separate the words with spaces. Instead, all of the text in a document
is run together and a language-specific dictionary is needed to find
word breaks. Lexbe does not have the ability to identify word breaks in
In some languages such as Arabic, the surrounding
context for a word (my, your, the, a, masculine/feminine, etc.) can be
expressed as characters added in front of or behind the word. For
example, "the apple" or "my apple" would not be two words but would be
different prefixes or suffixes added to "apple". To search for text in
these languages, adding a * in the front and back of the word will pick
up most of the variants, like this: *apple*.
The above discussion provides examples only, and there may be other limitations other than the ones
If you have other questions, please contact customer
support at LexbeSupport.com.