Fixed a bug where the precompiler would not work on .NET Framework editions of MSBuild, like in Visual Studio for Windows, when targeting frameworks greater than .NET 6.
Frequently used post-processors will no longer be optimized using dynamic code generation, because of concerns about the code generator's robustness. Performance might slightly regress. A more robust code generation system might be introduced in a future version of Farkle.
More methods in the builder check their arguments for null and throw ArgumentNullExceptions.
Some documentation was fixed.
The nullability annotations for some generic methods in the builder were fixed.
Added params in the array parameter overload of Nonterminal.CreateUntyped that takes a name and a series of production builders.
The color of the goto cells in the HTML tables was changed to a brighter shade of blue whose contrast is compliant with WCAG 2 AA.
C# users can now define single-member productions by directly calling Finish(Constant) without calling Appended() or Extended(). Previously only some combinations were supported (Finish on typed designtime Farkles and FinishConstant on literals), but now all can be used.
The HTML DFA States tables were restyled to make them consistent with the LALR States tables. In particular the columns were swapped and the dark grey background of the character cells was removed.
The farkle new command will place the output in the current directory instead of the input grammar's or assembly's directory, if not given an output path by the user.
The Production<T> type became a covariant interface, fixing issue #26. Because it used to be a sealed class with no members, it is not expected to be a breaking change. User code is not allowed to implement it.
Trimming warnings were reduced. A couple of them remain and might appear when building grammars without the precompiler, but they pose no risk, removing them was non-trivial.
Breaking change:Farkle.Tools.MSBuild require MSBuild 17, which comes with Visual Studio 2022 and the .NET 6 SDK.
Farkle supports precompiling .NET 6 assemblies from Visual Studio.
Fixed random compiler errors when using the F# production builder operators .>> and .>>..
The precompiler supports writing both a conflict report and detailed MSBuild errors via the FarklePrecompilerErrorMode property. Its allowed values are ReportOnly (default), ErrorsOnly and Both.
String regexes support specifying predefined sets by their property name as well. For example, until now you could match all characters in the Greek Extended character set by specifying \p{Greek Extended}. From now on, \p{GreekExtended} will work too.
Various bug fixes, particularly in the precompiler.
Minor breaking change:Farkle.Builder.RegexGrammar.designtime's type became a regular DesigntimeFarkle<Regex>, after being changed to a Nonterminal<Regex> in Farkle 6.3.0 by mistake. This change is unlikely to affect any regular user of Farkle.
Fixed a bug where Farkle's assembly was not shrunk enough when trimmed.
Minor breaking change: Text with legacy CR line endings is no longer supported. Since this version, they will no longer be recognized as line endings by position tracking. When Farkle's next major version gets released, they will cause lexical errors.
Minor breaking change: The Farkle.Builder.GrammarMetadata type has an additional field of type Farkle.Builder.OperatorPrecedence.OperatorScope. User code that does not directly set metadata to designtime Farkles is not affected.
The precompiler will now generate an HTML report if a grammar has LALR conflicts.
The precompiler now works on Visual Studio for Windows. In this case only, it requires to install the .NET tool Farkle.Tools.
When building a grammar, if Farkle cannot distinguish between two symbols, it will now include an example word that made it indecisive in the error message. A new build error type named IndistinguishableSymbols2 was introduced.
Any type of designtime Farkle can now be renamed or set metadata like case sensitivity, comments or operator scopes, without using the DesigntimeFarkle.cast function. There were some binary breaking changes but none is source breaking.
Operator scopes will now correctly recognize multiple representations of the same designtime Farkle, such as an original terminal and a renamed one, or a string, and a designtime Farkle created with the literal operator that got passed the same string.
Fixed a bug on the new CLI tool command where the --prop option could not be specified more than once.
Fixed a bug where the new CLI tool command could not be used on projects or assemblies.
Fixed a bug where titlecase letters were not excluded when building case-insensitive grammars.
Fixed a bug where the sepBy operator could not match exactly one item.
Breaking change: The Farkle.Builder.LALRBuildTypes module, as well as most functions of the Farkle.Builder.LALRBuild module became internal.
Minor breaking change: Whitespace inside the "between", "at least" and "exactly n times" regex string quantifiers is no longer allowed. Using string regexes like \d{ 4} will cause an error.
Minor breaking change: The Farkle.Builder.IRawDelegateProvider interface and a function in the Farkle.Builder.DFABuild module became private. They were public by accident and served no purpose for Farkle's users.
~~Minor breaking change: Users that write their own tokenizers must ensure that CRLF line endings are Advanced at once, otherwise the character stream's position will be incorrect. The vast majority of use cases (those that doesn't involve custom tokenizers) will not be affected by this change.~~ Reverted in Farkle 6.3.0.
Minor breaking change: Terminals, literals, and groups that are case-insensitively named NewLine will have an underscore prepended to their name when built, making them _NewLine for example. This change is a temporary workaround for a design deficiency of Farkle, where these terminals could end line groups and comments. It will be thoroughly fixed in the next major version. Because these names are only used for diagnostics and documentation generation, parser behavior will not be affected by this change. Nor will grammars read by EGT files be changed.
Minor breaking change: The \s and \S string regex symbol will now match exactly either horizontal tabs, line feeds, carriage returns or spaces. Other characters like vertical tabs and non-breaking spaces are no longer matched. This change matches Farkle's definition of whitespace. To revert to the previous behavior use \p{Whitespace} or \P{Whitespace}.
Farkle's string regexes got many improvements and bug fixes, bringing their syntax closer -but not a 100% match- to popular regex languages. Take a look at the string regex reference for more details.
The precompiler now works when used on a project targeting a framework earlier than .NET Core 3.1.
The Position.Advance method got a new overload that accepts a ReadOnlySpan of characters. It is recommended over the existing one that accepts a single character because it reliably handles line endings.
Building a grammar can now be cancelled by new functions introduced in Farkle.Builder's DFABuild, LALRBuild and DesigntimeFarkleBuild modules. Additionally the Build and BuildUntyped extension methods of designtime Farkles now accept an optional cancellation token.
Fixed a bug where the wrong error position was sometimes reported on text with LF line endings.
Fixed a bug where the wrong error position was reported on syntax errors.
The ParserApplicationExceptionWithPosition type was deprecated in favor of a new constructor in ParserApplicationException.
The EGT file reader will throw an exception if it tries to load a grammar file that shifts on EOF in one of its LALR states (which would cause infinite loops).
The designtime Farkles that are returned by the many and many1 operators now have the correct name.
Throwing a ParserException during post-processing will not be thrown in user code inside a PostProcessorException, but will be caught by the runtime Farkle API, allowing to customize the parser error message in greater detail.
Breaking change: The functions RuntimeFarkle.ofEGTFile and RuntimeFarkle.ofBase64Stringas well as their corresponding RuntimeFarkle static methods were removed. Users are advised to migrate away from GOLD Parser.
Breaking change: The RuntimeFarkle.GetBuildError method was replaced by GetBuildErrors, which returns a list of build errors.
Breaking change: The Farkle.Grammar.Grammar.Properties member now holds a strongly-typed record of informative grammar properties. Unrecognized GOLD Parser properties such as "Author", "Description" and "Version" are discarded. Existing grammar files remain compatible.
Breaking change: The members of Farkle's discriminated unions can no longer be accessed in F# using a method. For example, let foo (x: Terminal) = printfn "%s" x.Name becomes let foo (Terminal(_, name)) = printfn "%s" name. For C# users, duplicate properties like Terminal.name with a lowercase "n" were removed; they are replaced by their corresponding title-cased properties.
Breaking change: The members of many of Farkle's discriminated unions got meaningful names. C# code using the older names might break.
Breaking change: Removed support for generating legacy grammar skeleton files from MSBuild using the Farkle item.
Breaking change: The Farkle.Common.SetOnce type became internal.
Farkle's builder supports defining operator precedence and associativity to resolve conflicts. The quick start guide was updated.
Farkle supports virtual terminals -terminals that are not backed by the default tokenizer's DFA but created by custom tokenizers-, allowing for scenarios like indent-based grammars. An F# sample of an indent-based grammar was published.
Dynamic code generation will be applied to post-processors that are frequently used, in a fashion similar to .NET's tiered compilation, regardless of whether their designtime Farkle is precompilable.
Build error reporting is improved. More build errors will be reported at the same time, without having to fix one to show the next.
Stack overflows when building extremely complex designtime Farkles were either mitigated or will throw recoverable exceptions.
Breaking change: .NET Framework 4.5 is no longer supported. The library targets .NET Standard 2.0, .NET Standard 2.1 and .NET 5.
Breaking change: Logging parser events is no longer supported. The logging function was removed from the signatures of parser APIs.
Breaking change: The function LALRParser.parseLALR was renamed to LALRParser.parse.
Breaking change: The Farkle.Parser.OptimizedOperations type was made internal.
Breaking change: Transformers no longer take the position of a terminal, but an ITransformerContext type containing more information.
Breaking change: Farkle's exception types were refactored. They inherit from FarkleException.
Breaking change: Post-processor exceptions are wrapped inside a PostProcessorException and thrown to user code.
Breaking change: Only objects of type PrecompilableDesigntimeFarkle can be marked for precompilation.
Minor breaking change: The Transformers and Fusers properties were removed from the Farkle.Builder.GrammarDefinition type.
Farkle will now optimize precompilable designtime Farkles using dynamic code generation. This feature is only supported on .NET Standard 2.1 if the runtime supports dynamic code compilation.
Breaking change: Removed the legacy API for creating runtime Farkles from EGT files (the API with the transformers and fusers). EGT files are still supported (for now), but users are strongly urged to rewrite their grammars using Farkle.Builder, or implement the PostProcessor interface themselves (not recommended).
Breaking change: The PostProcessor type was moved to the root Farkle namespace. Some reusable post-processors were moved to the new Farkle.PostProcessors module.
You can now add lexical groups in a grammar. They resemble GOLD Parser's feature, but always advance by character and do not not support nesting.
Add a couple of methods in runtime Farkles to easily check whether building it had succeeded.
Lexical errors are reported at the point they occur; not at the point of the first character read by the tokenizer.
Made the untyped builder API easier to access. See the deprecation notice for the new functions to use.
Minor breaking change: The CharStream.readChar value gets the character index by value, not by reference. Callers have to increment it accordingly to get further characters.
Minor breaking change: The type Farkle.Grammar.OptimizedOperations was moved to the Farkle.Parser namespace; it was there for historical reasons.
As you might have seen, breaking changes on public members that do not affect the average Farkle user will not warrant a major version increase.
Farkle's speed more than doubled by disabling tailcall optimizations.
Minor breaking change: The API of the CharStream type slightly changed. Most notably, the type CharStreamIndex was removed in favor of uint64, and the order of the last two arguments in the function CharStream.read has changed.
The types DesigntimeFarkle<TResult> and PostProcessor<TResult> are covariant. The change was made possible by Covarsky, a tool written for this purpose.
Added a function called Regex.allButChars (Regex.NotOneOf for C#) that creates regexes accepting all characters except certain ones.
User code exceptions during post-processing are not captured anymore.
Farkle.Tools.MSBuild works with all .NET Core SDK versions after 2.0.
It happened. Farkle can create grammars without the need of GOLD Parser. Farkle is now a parsing library on its own.
Move the CharStream type in the Farkle.IO namespace.
Add methods to parse text from .NET TextReaders. They should be preferred over parsing .NET Streams because the latter are supposed to contain binary data, not text.
The Farkle.CSharp namespace is no longer required. C# users just have to use Farkle to get their extension methods, unless they are writing their own post-processors for GOLD Parser grammars, where they have to use Farkle.PostProcessor.CSharp.
Breaking change: Farkle.Tools.MSBuild was upgraded to .NET Core 3.1. Nothing significant changed though, which means those who still use .NET Core 2.1 can stay in a previous version.
Breaking change: Reading grammars from EGT files now raises an exception.
Breaking change: Some utility functions that had nothing to do with parsing were either removed or made internal.
Breaking change: Internal errors of the parser (in the unfortunate case they happen) throw an exception. In the next release, exceptions in a transformer or fuser will not be caught either.
Fix a bug where the tokenizer would erroneously report an EOF instead of a lexical error. - #8
Add CharStream.TryLoadFirstCharacter. With this method, you can check whether the input of a character stream has ended, and safely access CharStream.FirstCharacter.
Bring back Grammar.StartSymbol, but implement it correctly this time.
Speed-up the tokenizer by using an array that handles ASCII characters.
Remove Grammar.StartSymbol, as it was unreliable, and useless to Farkle.
Remove the OutputDirectorySuffix metadata introduced in 5.0.0-rc.5. Generated source files by Farkle.Tools.MSBuild are always added in the same directory as the project.
Fixed a bug which sometimes made Farkle.Tools.MSBuild unusable for C# users.
Added a new MSBuild item metadata called OutputDirectorySuffix which is useful for generating templates in a different directory than the EGT file.
Add a new benchmark which compares a Farkle-made JSON parser to Chiron, which is powered by FParsec. You can see the results of the benchmarks over here.
The CLI tool generates post-processor skeletons by default.
The CLI tool can automatically find the EGT file to process (if there is only one in the current directory), and the language (if there are C# or F# projects in the current directory).
Fixed a bug where generated source files would sometimes have duplicate production names in their enumerated type. - #7
Farkle now has better C# support. Just write using Farkle.CSharp;, and you are good to go!
Farkle now has a CLI tool helper. It can generate a grammar definition file that contains the terminal and production types for your grammar, as well as the EGT file in base-64 encoding. It can also create a skeleton source file to help you write a post-processor. What is more, it supports both C# and F#!
Farkle now has MSBuild integration. You can auto-generate a source file describing your grammar, and not have to carry EGT files around.
As always, performance was improved, especially in the EGT file reader.
Breaking change: In your post-processor, if you have functions like take2Of production (index1, index2) count func, remove the count parameter.
Breaking change: The Token type was moved to Farkle.Parser and is not needed by the AST type.
The functions for the RuntimeFarkle type are in their own module.
The reader for GOLD Parser EGT files is now based on a BinaryReader, and is faster and more lightweight.
The parser directly generates Abstract Syntax trees, eliminating the need to convert them from reductions, a very hard-to-use type.
The parser does not keep the parsing log messages. Instead, a callback that is provided by the user is called from which the message can be arbitrarily processed.
The code was yet again simplified and reorganized into different namespaces.
The library became much faster after performance profiling.
Overhauled the low-level parser API. It is a simple 5-line type called Parser. Its new design prevents misusing the API (like continuing parsing on a completed parser state), and decouples it from any implementation.
Also, the GOLDParser class is changed too. It is no more a static class, and its design encourages creating a grammar only once.
Polished the API; changed the names of some types (like Production.Nonterminal to Head), added prettier C# names and made some types internal.
And last yet not least... Everything is documented.