Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
586 lines (470 loc) · 23.2 KB

CONTRIBUTING.md

File metadata and controls

586 lines (470 loc) · 23.2 KB

Contributing

Quick start

Install Node and pnpm, following the version ranges declared in the engines field of the root package.json file.

If you work on multiple projects that require different versions of Node and pnpm, we recommend installing them in an isolated environement (e.g. with Conda) or with a specialised tool like Volta.

Then, run:

pnpm install
pnpm start

Development

  • pnpm start - start the H5Web stand-alone demo
  • pnpm start:storybook - start the component library's Storybook documentation site at http://localhost:6006

pnpm cheat sheet

  • pnpm install - install the dependencies of every project in the workspace and of the workspace itself
  • pnpm --filter <project-name> add [-D] <pkg-name> - add a dependency to a project in the workspace
  • pnpm [run] <script> [--<arg>] - run a workspace script
  • pnpm [run] "/<regex>/" [--<arg>] - run multiple workspace scripts in parallel
  • pnpm [run] --filter {packages/*} [--parallel] <script> [--<arg>] - run a script in every project in the packages folder
  • pnpm [exec] <binary> - run a binary located in node_modules/.bin (equivalent to npx <pkg-name> for a package installed in the workspace)
  • pnpm dlx <pkg-name> - fetch a package from the registry and run its default command binary (equivalent to npx <pkg-name>)
  • pnpm why -r <pkg-name> - show all project and packages that depend on the specified package
  • pnpm outdated -r - list outdated dependencies in the workspace
  • pnpm up -rL <pkg-name> - update a package to the latest version in every project

Dependency management

  1. Run pnpm outdated -r to list dependencies that can be upgraded.
  2. Read the changelogs and release notes of the dependencies you'd like to upgrade. Look for potential breaking changes, and for bug fixes and new features that may help improve the codebase.
  3. Run pnpm up -rL <pkg-name> to upgrade a dependency to the latest version in all projects. Alternatively, you can either replace -r with --filter to target specific projects, or edit the relevant package.json file(s) manually and run pnpm install (but make sure to specify an exact dependency version rather than a range - i.e. don't prefix the version with a caret or a tilde).

If you run into peer dependency warnings and other package resolution issues, note that pnpm offers numerous solutions for dealing with them, like pnpm.peerDependencyRules.allowedVersions.

DefinitelyTyped packages

The major versions of @types/* packages must be aligned with the major versions of the packages they provide types for—i.e. foo@x.y.z requires @types/foo@^x.

For convenience, some @types packages can be quickly upgraded to their latest minor/patch version by running pnpm up -r.

Workspace dependencies

To reference a workspace dependency, use pnpm's workspace protocol with the * alias - e.g. "@h5web/lib": "workspace:*". This tells pnpm to link the dependency to its corresponding workspace folder, and saves you from having to keep the version of the dependency up to date. During publishing, pnpm automatically replaces workspace:* with the correct version.

A workspace dependency's package.json must include a main field pointing to the dependency's source entry file - e.g. src/index.ts. This is the key to this monorepo set-up, as it avoids having to run watch tasks in separate terminals to automatically rebuild dependencies during development.

Obviously, a package's main field cannot point to its source TypeScript entry file once published, as consumers may not understand TypeScript. Additionally, package.json needs to point to more entry files (type declarations, ESM build, etc.) and do so in a way that is compatible with various toolchains (webpack 4, webpack 5, Parcel, Rollup, Vite, CRA, etc.) pnpm provides a nice solution to this problem in the form of the publishConfig field.

Icon set

H5Web uses the Feather icon set. Icons can be imported as React components from react-icons/fi.

CSS Modules

Styles are written with CSS Modules. Global styles should be avoided.

In most cases, a component that needs styling should come with its own CSS Modules file - e.g. MyComponent.module.css. If multiple components need to use the same style rules, they may import the same CSS Modules file:

import styles from './ThisComponent.module.css';
import otherComponentStyles from './OtherComponent.module.css';

However, it is better to write reusable components or take advantage of a CSS Modules feature called composition:

/* utils.module.css */
.baseBtn {
  color: green;
}

/* ThisComponent.module.css */
.thisBtn {
  composes: baseBtn from './utils.module.css';
  color: blue;
}

.selectedBtn {
  composes: thisBtn;
  color: red;
}

To avoid style ordering issues, modules that are composed from other modules (utils.module.css in the example above) should be used solely for composition and never be imported. In other words, you should never compose from a class located in the CSS Modules file of another React component:

/*
  >>> BAD <<<
  If `OtherComponent.module.css` gets imported after `ThisComponent.module.css,
  the `.baseBtn` rule will end up after `.thisBtn` and `color: blue` will be applied.
*/

/* OtherComponent.module.css */
.baseBtn {
  color: blue;
}

/* ThisComponent.module.css */
.thisBtn {
  composes: baseBtn from './OtherComponent.module.css';
  color: red;
}

/*
  >>> GOOD <<<
  Move `.baseBtn` to a utility module.
*/

/* utils.module.css */
.baseBtn {
  color: blue;
}

/* OtherComponent.module.css */
.otherBtn {
  composes: baseBtn from './utils.module.css';
}

/* ThisComponent.module.css */
.thisBtn {
  composes: baseBtn from './utils.module.css';
  color: red;
}

Build

  • pnpm build - build the H5Web stand-alone demo
  • pnpm build:storybook - build the component library's Storybook documentation site
  • pnpm serve - serve the built demo at http://localhost:5173
  • pnpm serve:storybook - serve the built Storybook at http://localhost:6006
  • pnpm packages - build packages (cf. details below)

Package builds

The build process of @h5web/lib works as follows:

  1. First, we use Vite to build the JS bundle of the library starting from the src/index.ts entrypoint. The dist/index.js bundle is then referenced from package.json. Vite also generates dist/style.css, which contains the compiled CSS modules that Vite comes across while building the React components.

  2. Second, we run build:dts to generate type declarations for TypeScript consumers. This is a two step process: first we generate type declarations for all TS files in the dist-ts folder with tsc, then we use Rollup to merge all the declarations into a single file: dist/index.d.ts, which is referenced from package.json. Note that since @h5web/shared is not a published package, it cannot be marked as an external dependency; its types must therefore be inlined into dist/index.d.ts, so we make sure to tell Rollup where to find them.

The build process of @h5web/app is the same with one exception: after Vite generates the styles of the app package, we concatenate them with the styles of the lib package so that consumers can just import a single stylesheet. This is taken care of by the build:css script thanks to a dedicated Vite config (vite.styles.config.js) and entrypoint (src/lib-styles.ts).

The build process of@h5web/h5wasm is also the same, but since the package does not include any styles, no styles.css file is generated.

Finally, since @h5web/shared is not a published package, it does not need to be built with Vite. However, its types do need to be built with tsc so that other packages can inline them in their own dist/index.d.ts.

Code quality

  • pnpm lint - run Prettier, ESLint and TypeScript on the entire workspace
  • pnpm lint:prettier - check that every file is formatted with Prettier
  • pnpm lint:eslint - lint every project with ESLint
  • pnpm [--filter <project-name|{folder/*}>] lint:eslint - lint specific projects
  • pnpm --filter <project-name> lint:eslint --inspect-config - inspect the linting configuration of a project
  • pnpm lint:root:eslint - lint files that don't belong to projects
  • pnpm lint:tsc - type-check every project with TypeScript
  • pnpm [--filter <project-name|{folder/*}>] lint:tsc - type-check specific projects
  • pnpm lint:root:tsc - type-check files at the root of the monorepo
  • pnpm lint:cypress:tsc - type-check the cypress folder
  • pnpm --filter @h5web/<lib|app> analyze - analyze a package's bundle (run only after building the package)
  • pnpm --filter storybook exec storybook doctor - diagnose problems with Storybook installation

Fixing and formatting

  • pnpm lint:prettier --write - format all files with Prettier
  • pnpm lint:eslint --fix - auto-fix linting issues in every project
  • pnpm [--filter <project-name|{folder/*}>] lint:eslint --fix - auto-fix linting issues in specific projects

Editor integration

Most editors support fixing and formatting files automatically on save. The configuration for VSCode is provided out of the box, so all you need to do is install the recommended extensions.

Testing

  • pnpm test - run unit, browser and providers tests with Vitest in watch mode (or once when on the CI)

Vitest is able to run on the entire monorepo thanks to the projects configuration in the root vitest.config.ts. It then uses each project's Vite or Vitest configuration to decide how to run the tests.

For the browser tests to work (*.browser.test.ts), a browser must first be installed with Playwright; see Browser tests section.

For the providers tests to work (h5grove-api.test.ts and h5wasm-api.test.ts), the sample HDF5 file must first be created and the h5grove support server must be running in a separate terminal; see pnpm support:* scripts and Providers tests section.

  • pnpm test:headless - use headless mode for browser tests, typically in the CI
  • pnpm test[:headless] run - run unit and browser tests once
  • pnpm test[:headless] [run] <filter> - run tests matching the given filter
  • pnpm test[:headless] --project <project-name> - run Vitest on a specific project
  • pnpm test[:headless] --coverage measure coverage (environment variable VITE_TEST_BROWSER must be set to chromium in .env.test.local and this browser must be installed with Playwright)
  • pnpm test[:headless] --ui [--coverage] - run tests in Vitest UI (with or without coverage)
  • pnpm support:setup - create/update Poetry environments required for testing the providers
  • pnpm support:sample - create sample.h5
  • pnpm support:h5grove - start h5grove support server

We also use Cypress to run end-to-end and visual regression testing:

  • pnpm cypress - open the Cypress end-to-end test runner (local dev server must be running in separate terminal)
  • pnpm cypress:run - run end-to-end tests once (local dev server must be running in separate terminal)

Browser tests

The @h5web/app package includes tests written for Vitest's Browser Mode. They are located under src/__tests__ and are suffixed with *.browser.test.ts. Each file covers a particular subtree of components of H5Web.

For the browser tests to work, you must first install the browser specified in environment variable VITE_TEST_BROWSER with Playwright:

pnpm exec playwright install <firefox,chromium,...>

H5Web's browser tests typically consist in rendering the entire app with mock data (i.e. inside MockProvider), executing an action like a real user would (e.g. clicking on a button, pressing a key, etc.), and then expecting something to happen in the DOM as a result. Most tests perform multiple actions and expectations consecutively to minimise the overhead of rendering the entire app again and again.

Loading interfaces

To allow developing and testing loading interfaces, as well as features like cancel/retry, MockProvider adds an artificial delay of 3s (SLOW_TIMEOUT) to some requests, notably to value requests for datasets prefixed with slow_.

Here is an example of how to test a loading interface in Vitest's Browser Mode, while also avoiding the long timeout:

// Mock the `delay` utility used by `MockProvider` (i.e. keep promises in pending state until we're ready to resolve them)
const runAll = mockDelay();

// Render the app but don't wait for loaders to disappear (since the promises are pending, we would wait indefinitely)
await renderApp({ waitForLoaders: false });

// Test the loading interface
await expect.element(page.getByText('Loading data...')).toBeVisible();

// Run all mock delays (i.e. resolve pending promises)
runAll();

// Test the final interface
await expect.element(page.getByRole('figure')).toBeVisible();

Debugging

You can use Vitest's debug utility to log a given DOM element or locator anywhere in your tests:

utils.debug(page.getByRole('figure'));

To ensure that the entire DOM is printed out in the terminal, environment variable DEBUG_PRINT_LIMIT has to be set to a large value when calling pnpm test (cf. .env.test and your own .env.test.local).

Providers

Two data providers are currently tested through their respective APIs: H5GroveApi and H5WasmApi. Each API test (<provider>-api.test.ts) works as follows:

  1. It instanciates the API using a sample file called sample.h5, located in support/sample/dist, that contains a lot of HDF5 datasets of various shapes and types.
  2. It retrieves the values of all the datasets in the sample file and stores them in an object.
  3. It takes a snapshot of that object and compares it to the existing snapshot (<provider>-api.test.ts.snap).
  4. If the new snapshot is the same, the test succeeds; if the new snapshot differs, the test fails and the differences are shown.

To get up and running with testing the providers locally, we recommend using pyenv and Poetry:

  1. Install the Python version specified in .python-version with pyenv: 
    pyenv install
  2. Install Poetry with pipx: 
    pipx install poetry
  3. Create the Poetry environements: 
    pyenv exec pnpm support:setup

Once the Poetry environments are created, you can create sample.h5, start h5grove and run the API tests:

pyenv exec pnpm support:sample
pyenv exec pnpm support:h5grove
pnpm test api

If the Python version specified in .python-version is globally available on your system, you may run the scripts above without pyenv exec.

If you need to intervene on the support projects and the Python version specified in .python-version is not globally available, make sure to run all Poetry commands from the root of the monorepo as follows:

```bash
pyenv exec poetry -C support/<project> <cmd>
pyenv exec poetry -C support/<project> add <dep>
pyenv exec poetry -C support/<project> run python <script>.py
```

If you're unable to create the sample file (for instance because your environment lacks support for float128), you may download it from silx.org and place it into the support/sample/dist folder. However, please beware that the file may not be up to date.

Visual regression

Cypress is used for end-to-end testing but also for visual regression testing. The idea is to take a screenshot (or "snapshot") of the app in a known state and compare it with a previously approved "reference snapshot". If any pixel has changed, the test fails and a diff image highlighting the differences is created.

Taking consistent screenshots across platforms is impossible because the exact rendering of the app depends on the GPU. For this reason, visual regression tests are run only on the CI. This is done by running cypress with the takeSnapshots feature flag: pnpm cypress:run --expose takeSnapshots=true.

Visual regression tests may fail in the CI, either expectedly (e.g. when implementing a new feature) or unexpectedly (when detecting a regression). When this happens, the diff images and debug screenshots that Cypress generates are uploaded as artifacts of the Lint & Test workflow, which can be downloaded and reviewed.

If the visual regressions are expected, the version-controlled reference snapshots can be updated by manually running the Lint & Test workflow on the working branch and by ticking the "Update Cypress snapshots" checkbox. Once Cypress has updated the reference snapshots, the workflow automatically opens a PR to merge the new and/or updated snapshots into the working branch. After this PR is merged, the visual regression tests in the working branch succeed and the branch can be merged into main.

Here is the summarised workflow:

  1. Push your working branch but don't open a PR yet.
  2. If the e2e job of the Lint & Test CI workflow fails, check out the logs.
  3. If the fail is caused by a visual regression (i.e. if a test fails on a cy.matchImageSnapshot() call), download the workflow's artifacts.
  4. Review the snapshot diffs. If the visual regression is unexpected: fix the bug, push it and start from step 2 again. If the visual regression is expected: continue to step 5.
  5. Run the Lint & Test workflow manually on your working branch, making sure to tick the "Update Cypress snapshots" checkbox.
  6. Wait for the workflow to complete, then go to the newly opened PR titled Update Cypress reference snapshots.
  7. Review the new reference snapshots once more and merge the PR (ideally with a rebase).
  8. Confirm that the Lint & Test workflow now succeeds on your working branch, then open a PR.

After step 4, it's also possible to commit/push the updated snapshots manually instead of going through steps 5 to 7.

Deployment

Note that the version of pnpm that Netlify installs by default is outdated and incompatible with this monorepo. We use the packageManager entry in the root package.json to specify a more recent version.

Release process

To release a new version and publish the packages to NPM:

  1. Check out main and pull the latest changes.
  2. Make sure your working tree doesn't have uncommitted changes and that the latest commit on main has passed the CI.
  3. Run pnpm version [ patch | minor | major | <new-version> ]

The pnpm version command:

  1. bumps the version in the workspace's package.json;
  2. copies the new version into each package's package.json (via the version script);
  3. commits and tags the changes, and then pushes the new commit and the new tag to the remote repository (via the postversion script).

This, in turn, triggers the Publish packages and Deploy Storybook workflows on the CI, which builds and publishes the packages to NPM (with pnpm -r publish) and deploys the Storybook site.

Note that pnpm publish modifies package.json by merging in the content of the publishConfig field.

Once the CI workflows have run successfully:

  • Make sure the new package versions are available on NPM and that the live Storybook site still works as expected.
  • Upgrade and test the packages in apps and code sandboxes, as required.
  • Write and publish release notes on GitHub.

Beta testing

The beta release process described below allows publishing packages to NPM with the next tag (instead of the default latest tag) so they can be beta-tested before the official release.

  1. Follow steps 1 and 2 of the normal release process.
  2. At step 3, run pnpm version <x.y.z-beta.0> (incrementing the beta version number as needed).

The CI will then build and deploy the packages with pnpm publish --tag next. Once the Publish packages workflow has run successfully, check that the beta packages are published with the correct tag by running npm dist-tag ls @h5web/lib. This command should print something like:

latest: <a.b.c>
next: <x.y.z>-beta.0

You can then install the beta packages with npm install @h5web/lib@next or the like and make sure that they work as expected. Once you're done testing, follow the normal release process, making sure to run pnpm version <x.y.z> at step 3 (without the beta suffix).

Once you've completed the release process, you may remove the next tag from the obsolete beta packages by running npm dist-tag rm @h5web/lib@<x.y.z-beta.0> next

Local testing

To test a package locally in another project without publishing it to NPM, follow these steps:

  1. Run pnpm packages.
  2. Navigate to the package's directory - e.g. cd packages/app.
  3. Run pnpm pack to pack the package into a tarball, optionally passing a target directory for the tarball with --pack-destination <dir>.
  4. Navigate to the project in which you want to install and test the package.
  5. Install the tarball with the project's package manager (e.g. pnpm add <path-to-tarball>).