Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
116 lines (76 loc) · 7.86 KB

CONTRIBUTING.md

File metadata and controls

116 lines (76 loc) · 7.86 KB

Contributing to GritQL

We welcome contributions in the form of pull requests and issues.

Note that this codebase isn't yet extensively documented. If you get stuck, please ask for help on Discord.

Development Setup

A high-level overview of tools you need to have installed:

  • Rust toolchain: for compiling the codebase. You'll need rustc v1.74 or newer.
    • To create WASM builds, run rustup target install wasm32-unknown-unknown.
  • C/C++ compiler. macOS: Xcode Command Line Tools via xcode-select --install, Linux: gcc, Windows: Microsoft Visual C++.
  • Emscripten: a C/C++ compiler toolchain for WASM. Install v3.1.56 with emsdk.
  • Node.js runtime: node, npm, npx are used to generate parsers from grammar.js files. You'll need node v18.5.0 or newer.
  • Yarn package manager. You'll need yarn (classic). Install v1.22.19 with npm install --global yarn.
  • Tree-Sitter CLI: provides tree-sitter binary for testing grammars. Install v0.22.2 with npm install --global tree-sitter-cli.
  • Terraform CLI. Install terraform with brew tap hashicorp/tap && brew install hashicorp/tap/terraform.
  • For snapshot testing review, you should install the cargo insta plugin: curl -LsSf https://insta.rs/install.sh | sh (https://insta.rs/docs/cli/)

Building the Code

Use git to clone this repository into a location of your choice.

git clone https://github.com/getgrit/gritql.git

Change into the cloned repository and make sure all submodules are correctly set up, including any nested submodules:

cd gritql
git submodule update --init --recursive

Before making any changes to the code, make sure you can run the tests and everything is initially passing:

cargo test --workspace

Feature Flags

We use feature flags to control which parts of the codebase are compiled.

Note that some proprietary server-only integrations are hidden behind the "server" feature flag. This flag is disabled by default, and code should compile without any additions.

For major changes, we put new features should be put into the grit_alpha feature flag. Features that are ready for broad release should be put into the grit_beta feature flag. This is used for all public releases.

Features that should be tested in CI should be put into the grit_ci feature flag. This is used for all CI tests.

Language Grammars

If GritQL is failing to match a code snippet, this can typically be fixed simply by adjusting the metavariable grammar for the target language.

Metavariable grammars are found under ./resources/metavariable-grammars. Typical fixes include:

  • Adding a new named field for a relevant node you want to manipulate.
  • Adding a grit_metavariable node as a choice in the corresponding spot where you want to substitute the metavariable.
  • Check this guide to debug grammars generally.

After making your changes, run the ./resources/edit_grammars.mjs script to regenerate the matching grammar. You can specify the language you want to regenerate to speed up the process.

For example:

node ./resources/edit_grammars.mjs yaml

Snippet contexts

Snippet contexts help when a snippet is a valid AST subtree, but needs to be in a larger tree to parse. For example, matching on a table name like $schema.$table in SQL is not valid SQL by itself, only when surrounded by something like SELECT x from $schema.$table is the snippet valid.

Snippet contexts are defined by implementing the snippet_context_strings method in the Language trait. This method returns a list of strings that are used to match the snippet in the larger tree. For example, the SQL implementation returns ["SELECT 1 from ", ";"] to match a table name in a SQL query.

Adding a New Target Language

Note: Grit involves two languages:

  • GritQL is our query language for searching and transforming codebases.
  • The “target language” is the language we are transforming (ex. Python). This document describes the process of adding new target languages to Grit.

Most of these steps involve iteration over a set of sample programs to get closer to correctness. The initial work for a language can typically be done in a day or two.

Here are the steps for adding a new target language:

  1. Add the language as a supported language in the GritQL grammar, like this.
  2. Find a tree sitter grammar for the language and add it as a submodule under resources/language-submodules.
  3. Add a simple parse test in crates/core/src/test.rs to ensure that the grammar is working.
  4. Copy the grammar file into resources/metavariable-grammars. This alternative grammar is used for parsing snippets in GritQL.
  5. Patch the metavariable grammar to include $.grit_metavariable anywhere we want to substitute a metavariable. This is usually at least $identifier and $literal.
    • For a snippet to match, it also needs to be a field. Often you’ll want to wrap $thing like: field('thing', choice($.grit_metavariable, $thing))
  6. Add a new language implementation in crates/core/languages/src. This involves implementing the Language trait and adding a new Language enum variant.
  7. Add snippet_context_strings like this to provide context for snippets to match in.
  8. Add test cases for the language in crates/core/src/test.rs. This is a good time to add a few dozen test cases to ensure that the language is parsed correctly, and that the metavariable grammar is working.

Internal steps

These steps are done in our cloud environment and are not necessary for contributors to do.

Troubleshooting

Tests failing due to pattern errors

Many tests depend on the standard library. If you see unexpected failures, you might have an outated copy of the standard library.

Try deleting target/.grit and running cargo test --workspace again.