One thing you’ll notice from the tokenizer’s output is that it’s flat. It does this character sequence recognition task until it reaches the end of the file. If the “For:” line contained any extra whitespaces, the tokenizer would silently strip those. In the above example, it recognizes a string of characters that start with “For: “ followed by a date format and creates a TODOLIST token based on that string. So what the tokenizer does is take the source file and create tokens based on the sequence of characters it encounters.
Let’s say we have the following source file written in some specialized language called “ToDo”: This becomes much easier to understand with an example. The difference between tokenizers and parsers is in the level that they operate. Tokenizers and parsers actually have very similar functions: they take a set of “things” (characters, tokens, etc.) and group, mark, or organize them according to the rules of the language.
Now, there are whole courses in schools dedicated to the discussion and understanding of lexers and parsers, but I will try to explain them in my own simplistic way that may be insufficient for designing more complex languages, but will be enough for the purposes of this blog post. You will see this mess in one of my prototypes. You can try and combine these into a single step, but I find that it results in really messy code. To do this, we need to go through a 2-step process: lexing (aka tokenizing) and parsing. In my case, I created a compiler that transforms BA Speak and QA Speak source files into Ruby code. For example, a typical C compiler transforms your.
0 kommentar(er)
