| Safe Haskell | None |
|---|---|
| Language | GHC2024 |
Circuit.Parser.Lexer
Description
Fast imperative lexers over ByteString.
These are complementary to the parser-combinator approach in
Circuit.Parser: they trade compositionality for raw speed
(zero-copy slices, "unsafeIndex" loops). The markup lexer can
also be viewed as a state machine (see oAccumStep / stepMarkupS)
which morphs cleanly into a Circuit when composition is needed.
Synopsis
- runWordLexerBS :: ByteString -> [ByteString]
- wordFreqBS :: ByteString -> [(ByteString, Int)]
- data MarkupCtx
- data MarkupToken
- runMarkupLexerBS :: ByteString -> [MarkupToken]
- runMarkupStateBS :: ByteString -> [MarkupToken]
- data ByteClass
- classifyByte :: MarkupCtx -> Word8 -> ByteClass
- data AccState = AccState {}
- accumStep :: AccState -> ByteClass -> Int -> (Maybe (ByteString -> MarkupToken, Int, Int), AccState)
- data WI = WI !Word8 !Int
- data OAccState = OAccState {}
- initOAccState :: OAccState
- oAccumStep :: OAccState -> (ByteClass, Int, MarkupCtx) -> (Maybe (ByteString -> MarkupToken, Int, Int), OAccState)
- stepMarkupS :: WI -> OAccState -> (Maybe (ByteString -> MarkupToken, Int, Int), OAccState)
Word lexing
runWordLexerBS :: ByteString -> [ByteString] Source #
Run word lexer directly over a ByteString.
>>>runWordLexerBS (C.pack "Hello, world!")["hello","world"]
wordFreqBS :: ByteString -> [(ByteString, Int)] Source #
Word frequency list — single pass, accumulate (word, count) pairs.
Markup lexing
data MarkupToken Source #
Markup token. ByteString fields are zero-copy slices of the input.
Constructors
| TOpenTag ByteString | |
| TCloseTag ByteString | |
| TContent ByteString | |
| TSelfClose | |
| TTagEnd | |
| TAttrName ByteString | |
| TAttrVal ByteString | |
| TComment ByteString |
Instances
| Eq MarkupToken Source # | |
Defined in Circuit.Parser.Lexer | |
| Show MarkupToken Source # | |
Defined in Circuit.Parser.Lexer Methods showsPrec :: Int -> MarkupToken -> ShowS # show :: MarkupToken -> String # showList :: [MarkupToken] -> ShowS # | |
runMarkupLexerBS :: ByteString -> [MarkupToken] Source #
Run markup lexer directly over a ByteString.
>>>runMarkupLexerBS (C.pack "<p>hi</p>")[TOpenTag "p",TContent "hi",TCloseTag "p"]
runMarkupStateBS :: ByteString -> [MarkupToken] Source #
Run via Traced (->) State pipeline.
The state-machine view is: each input byte is classified (classifyByte),
paired with its offset and the current context, and fed to oAccumStep.
stepMarkupS compiles this to WI -> OAccState -> (Maybe ..., OAccState),
the shape of a single step in a traced iteration. runMarkupStateBS
unwinds that step over the input.
Byte classification
Accumulator state
Accumulator: track start offset and length of current token being built.
accumStep :: AccState -> ByteClass -> Int -> (Maybe (ByteString -> MarkupToken, Int, Int), AccState) Source #
Step the accumulator given a byte class, current context, and byte index.
Traced / state-machine interface
Offset-tracking accumulator state for the Traced pipeline.
initOAccState :: OAccState Source #
Initial accumulator state.
oAccumStep :: OAccState -> (ByteClass, Int, MarkupCtx) -> (Maybe (ByteString -> MarkupToken, Int, Int), OAccState) Source #
Step the offset accumulator.
stepMarkupS :: WI -> OAccState -> (Maybe (ByteString -> MarkupToken, Int, Int), OAccState) Source #
Compiled step function.