Introduction

Rexer is a simple regex lexer that is easy to use.

This is a documentation on how to use Rexer and how to not fail your projects.

How to Get Rexer?

You can get Rexer through NuGet or building it yourself

I will use the NuGet version however as it is easier to get than building it.

First of all, to install NuGet you need to use the Install-Package Rexer command.

Then add using Rexer; at the top of your code.

Starting off...

First of all, define your test code, I will be using "HelloWorld"

Then I will make a new Lexer object with Lexer _lexer = new Lexer();

Your current code should look like this:


                string _code = "\"HelloWorld\"";
                Lexer _lexer = new Lexer();

Afterwards we want to create a lexer definition for a string. We will use the AddLexerDefintion function.


                string _code = "\"HelloWorld\"";
                Lexer _lexer = new Lexer();
                _lexer.AddLexerDefintion(new LexerDefinition("\\\"([^']+)\\\"", TokenType.STR));

But before all of that we have to create our TokenType enum.

Make an enum that derives from byte

Add your types.

Make an unknown token and set it to 255.


                enum TokenType : byte
                {
                STR,
                UNK = 255
                }

Now you should be fine, just change the LexerDefintion token type to match yours.

Now you have to set the Lexer to use your unknown token type definition

_lexer.UnknownToken = TokenType.UNK;

The Finishing Touches.

Now all you have to do is run the doString function with the text, and output it as a 'List'


                string _code = "\"HelloWorld\"";
                Lexer _lexer = new Lexer();
                _lexer.AddLexerDefintion(new LexerDefinition("\\\"([^']+)\\\"", TokenType.STR));
                List< Token > _tokens = _lexer.doString(_code);
                //And to test the code.
                foreach(Token t in _tokens)
                {
                    Console.WriteLine($"Type: {t._type}, Defined As: {t._def}");
                }

End.

Now you should've turned your string into tokens, if there was a problem with the code, open and issue on the github, and someone will surely help.