This module contains all classes for lexing LaTeX code, as well as general purpose base classes for incremental LaTeX decoders and encoders, which could be useful in case you are writing your own custom LaTeX codec.
A collections.namedtuple() storing information about a matched token.
See also
The name of the token as a str.
The matched token text as bytes. The constructor also accepts text as memoryview, in which case it is automatically converted to bytes. This ensures that the token is hashable.
Returns the decoded token text in the specified encoding.
Note
Control words get an extra space added at the back to make sure separation from the next token, so that decoded token sequences can be str.join()ed together.
For example, the tokens b'\hello' and b'world' will correctly result in u'\hello world' (remember that LaTeX eats space following control words). If no space were added, this would wrongfully result in u'\helloworld'.
Bases: codecs.IncrementalDecoder
A very simple lexer for tex/latex code.
Yield tokens without any further processing. Tokens are one of:
Set state. The state must correspond to the return value of a previous getstate() call.
List of token names, and the regular expressions they match.
Bases: latexcodec.lexer.LatexLexer
A very simple incremental lexer for tex/latex code. Roughly follows the state machine described in Tex By Topic, Chapter 2.
The generated tokens satisfy:
Bases: latexcodec.lexer.LatexIncrementalLexer
Simple incremental decoder. Transforms lexed LaTeX tokens into unicode.
To customize decoding, subclass and override get_unicode_tokens().
Decode LaTeX bytes_ into a unicode string.
This implementation calls get_unicode_tokens() and joins the resulting unicode strings together.
Decode every token in inputenc encoding. Override to process the tokens in some other way (for example, for token translation).
Input encoding. Must extend ascii.
Bases: codecs.IncrementalEncoder
Simple incremental encoder for LaTeX. Transforms unicode into bytes.
To customize decoding, subclass and override get_latex_bytes().
Encode the unicode_ string into LaTeX bytes.
This implementation calls get_latex_bytes() and joins the resulting bytes together.
Encode every character in inputenc encoding. Override to process the unicode in some other way (for example, for character translation).
Input encoding. Must extend ascii.