| [ Index ] |
PHP Cross Reference of DokuWiki |
[Source view] [Print] [Project Stats]
Lexer adapted from Simple Test: http://sourceforge.net/projects/simpletest/ For an intro to the Lexer see: https://web.archive.org/web/20120125041816/http://www.phppatterns.com/docs/develop/simple_test_lexer_notes
| Author: | Marcus Baker http://www.lastcraft.com |
| File Size: | 642 lines (24 kb) |
| Included or required: | 0 times |
| Referenced: | 0 times |
| Includes or requires: | 0 files |
Lexer:: (22 methods):
__construct()
addPattern()
addEntryPattern()
addCloserPattern()
addExitPattern()
addSpecialPattern()
mapHandler()
parse()
getModeStack()
dispatchTokens()
isModeEnd()
isSpecialMode()
decodeSpecial()
invokeHandler()
reduce()
canEnter()
nearestGuardedAncestor()
closerPosition()
opaqueSpans()
verbatimExit()
resetCloserMemos()
escape()
| __construct($handler, $start = "accept", $case = false) X-Ref |
| Sets up the lexer in case insensitive matching by default. param: Handler $handler Handling strategy by reference. param: string $start Starting handler. param: boolean $case True for case sensitive. |
| addPattern($pattern, $mode = "accept") X-Ref |
| Adds a token search pattern for a particular parsing mode. The pattern does not change the current mode. param: string $pattern Perl style regex, but ( and ) param: string $mode Should only apply this |
| addEntryPattern($pattern, $mode, $new_mode) X-Ref |
| Adds a pattern that will enter a new parsing mode. Useful for entering parenthesis, strings, tags, etc. param: string $pattern Perl style regex, but ( and ) lose the usual meaning. param: string $mode Should only apply this pattern when dealing with this type of input. param: string $new_mode Change parsing to this new nested mode. |
| addCloserPattern($pattern, $mode, $boundary = null) X-Ref |
| Requires a closer to exist ahead before any entry pattern may enter the given mode. Whenever an entry pattern for $mode matches, the subject is scanned from the end of that match for $pattern — before the next $boundary match if a boundary is given, anywhere ahead otherwise. If no closer is found the entry is rejected and its delimiter stays literal text; see reduce() for how the rejected match is discarded. Keeping the check out of the entry pattern is what makes it affordable. A closer lookahead written into the entry pattern is re-evaluated for every candidate the regex engine tries — quadratic on input dense with delimiters that never close. Here the scan runs once per position and its verdict is memoized, so together with the lexer consuming each entered span the whole parse stays linear; see CloserPattern for the scan and its memo. The pattern must match where the closing delimiter starts, with flanking requirements expressed as lookarounds (the convention exit patterns follow, e.g. (?<=[^\s])\*\* for strong). A pattern that consumed flanking context instead would skew the closer positions canEnter() compares across modes, and could not see a closer whose delimiter directly follows the opener, since the scan starts only after the entry match. The closer must not be able to match where the boundary matches, since the scan stops unconditionally at the boundary. It also does not look inside content the lexer consumes atomically; see opaqueSpans(). A mode has exactly one closer check, shared by all its entry patterns; registering another replaces it. The memo is reset at the start of every parse() run. return: void param: string $pattern regex fragment matching the closing delimiter, param: string $mode the mode entered by the guarded entry patterns param: string|null $boundary regex fragment the closer must occur |
| addExitPattern($pattern, $mode) X-Ref |
| Adds a pattern that will exit the current mode and re-enter the previous one. param: string $pattern Perl style regex, but ( and ) lose the usual meaning. param: string $mode Mode to leave. |
| addSpecialPattern($pattern, $mode, $special) X-Ref |
| Adds a pattern that has a special mode. Acts as an entry and exit pattern in one go, effectively calling a special parser handler for this token only. param: string $pattern Perl style regex, but ( and ) lose the usual meaning. param: string $mode Should only apply this pattern when dealing with this type of input. param: string $special Use this mode for this one token. |
| mapHandler($mode, $handler) X-Ref |
| Adds a mapping from a mode to another handler. param: string $mode Mode to be remapped. param: string $handler New target handler. |
| parse($raw) X-Ref |
| Splits the page text into tokens. Will fail if the handlers report an error or if no content is consumed. If successful then each unparsed and parsed token invokes a call to the held listener. return: boolean True on success, else false. param: string $raw Raw HTML text. |
| getModeStack() X-Ref |
| Gives plugins access to the mode stack return: StateStack |
| dispatchTokens($unmatched, $matched, $mode, $initialPos, $matchPos) X-Ref |
| Sends the matched token and any leading unmatched text to the parser changing the lexer to a new mode if one is listed. return: boolean False if there was any error from the parser. param: string $unmatched Unmatched leading portion. param: string $matched Actual token match. param: bool|string $mode Mode after match. A boolean false mode causes no change. param: int $initialPos param: int $matchPos Current byte index location in raw doc thats being parsed |
| isModeEnd($mode) X-Ref |
| Tests to see if the new mode is actually to leave the current mode and pop an item from the matching mode stack. return: boolean True if this is the exit mode. param: string $mode Mode to test. |
| isSpecialMode($mode) X-Ref |
| Test to see if the mode is one where this mode is entered for this token only and automatically leaves immediately afterwoods. return: boolean True if this is the exit mode. param: string $mode Mode to test. |
| decodeSpecial($mode) X-Ref |
| Strips the magic underscore marking single token modes. return: string Underlying mode name. param: string $mode Mode to decode. |
| invokeHandler($content, $state, $pos) X-Ref |
| Dispatches a token to the handler. Resolves mode name aliases (e.g. unformattedalt → unformatted) and delegates all dispatch logic to Handler::handleToken(). return: bool param: string $content Text parsed. param: int $state One of the DOKU_LEXER_* constants identifying the param: int $pos Current byte index location in raw doc |
| reduce($raw, $offset) X-Ref |
| Tries to match the next token starting at `$offset` in `$raw`. The full subject is passed to the regex engine (rather than a truncated tail) so that lookbehind assertions in the registered patterns can see characters before the current offset. Empty subjects (offset past end) will not be matched. A matched entry pattern for a guarded mode (one with a closer pattern) is discarded when canEnter() rejects it: its delimiter stays unparsed text and matching resumes one byte on, so a shorter delimiter overlapping the rejected one still gets its turn. Resuming past the delimiter also skips any rival pattern anchored at that exact byte, which is safe: guarded modes are registered only by the core, and two core delimiters sharing a byte share an equivalent closer, so one rejection implies the other. Plugin patterns are never guarded and so never take this path. return: array|bool Three item list of unparsed content followed by the param: string $raw The full subject to parse. param: int $offset Byte offset at which to resume matching. |
| canEnter(string $mode, string $subject, int $from) X-Ref |
| May an entry pattern for the given mode enter at this position? Two conditions must hold. First, a valid closer for the mode must exist ahead, before its boundary — otherwise a delimiter that can never close would stay open forever. Second, when the entry sits inside a guarded mode, that mode's closer must not come first: an inner delimiter whose closer lies beyond the enclosing closer can never close within its parent, so it stays literal rather than span across the parent boundary (e.g. a stray '*' in ''glob/*.conf'' pairing with the '*' of a following ''…'' span). The enclosing mode is the nearest guarded ancestor on the stack, not necessarily the immediate parent; see nearestGuardedAncestor(). return: bool param: string $mode the mode entered by the matched entry pattern param: string $subject the full subject being lexed param: int $from byte position just after the entry pattern match |
| nearestGuardedAncestor(string $mode) X-Ref |
| The nearest mode on the stack that has its own closer and could thus constrain where a delimiter entering $mode may close, or null if none does. The search walks from the immediate parent outward, stepping over unguarded modes — plugins, list items, anything with no closer. Such a mode offers no closer to compare against, yet a guarded ancestor beyond it still constrains the inner delimiter (e.g. ''strong'' around a plugin span holding an ''emphasis'' whose only closer lies further on), so skipping it reaches that ancestor. Only the nearest guarded ancestor matters: when it opened it was validated against its own nearest guarded ancestor, so the "closes before its parent" relation chains up the stack and one level is enough. The walk also stops at $mode itself — a same-mode ancestor shares this candidate's closer pattern, so its closer cannot fall before the candidate's and can never be the rejecting constraint. return: string|null the nearest guarded ancestor, or null if none param: string $mode the mode about to be entered |
| closerPosition(string $mode, string $subject, int $from) X-Ref |
| Byte position where the first valid closer for the given mode at or after $from starts, or null if none exists; delegates to the mode's CloserPattern (see position()). The scan regex is compiled on first use, not in addCloserPattern(), because the opaque-span derivation needs the patterns of all connected modes, and other modes' connectTo() calls may run after this mode's postConnect() has registered the closer. return: int|null param: string $mode the mode whose closer pattern applies param: string $subject the full subject being lexed param: int $from byte position just after the entry pattern match |
| opaqueSpans(string $mode) X-Ref |
| Derives the spans within the given mode whose content a closer scan must not look into. Content the lexer consumes without exposing it to the mode's exit pattern can never hold a real closer — a %%..%% span may contain the characters that would close an enclosing bold span, yet the bold exit cannot fire inside it. Such content is identified from the already registered patterns: - A plain or special pattern is consumed in one step, so the pattern itself describes the span. (Zero-width matches would stall parse(), so each consumes at least one byte.) - An entry pattern leads into a nested mode. If that mode is verbatim (see verbatimExit()), the lexer consumes up to its first exit, so the span is the entry pattern, a lazy body, and the exit. Other nested modes have no statically known extent — their content is scanned as plain text, leaving the closer check an approximation there. return: string[] regex fragments, each matching one whole span param: string $mode the mode whose closer scan needs the spans |
| verbatimExit(string $mode) X-Ref |
| The exit pattern of the given mode if the mode is verbatim, null otherwise. A mode is verbatim when its pattern set consists solely of exit patterns (e.g. nowiki): nothing can match inside it, so it consumes everything up to its first exit match. Several exits are combined into an alternation. return: string|null regex fragment matching the mode's exit param: string $mode the mode entered by an entry pattern |
| resetCloserMemos() X-Ref |
| Forgets all closer scan verdicts. Called at the start of every parse() run, since the memos only hold for the subject they were computed on. return: void |
| escape($str) X-Ref |
| Escapes regex characters other than (, ) and / return: string param: string $str |