GitHub Action for Continuous Benchmarking

This repository provides a GitHub Action for continuous benchmarking. This action depends on nothing other than GitHub; it collects data from benchmark outputs and writes them to GitHub pages branch while GitHub Action workflow. You can see the visualized benchmark results in the GitHub pages of your project.

This action currently supports

• cargo bench for Rust projects
• go test -bench for Go projects
• benchmark.js for JavaScript/TypeScript projects
• pytest-benchmark for Python projects with pytest

Multiple languages in the same repository is supported for polyglot projects.

Japanese Blog post

Example projects for each languages are in examples/ directory. Live example workflows are in .github/workflows/ directory. Workflow actions are:

• Rust:
• Go:
• JavaScript:
• Python (pytest-benchmark):

All benchmark charts from above workflows are gathered in GitHub pages:

https://rhysd.github.io/github-action-benchmark/dev/bench/

Mouse over on data point shows a tooltip. It includes

• Commit hash
• Commit message
• Date and committer
• Benchmark value

Clicking data point in chart opens the commit page on GitHub repository.

At bottom of the page, download button is available for downloading benchmark results as JSON file.

Since performance is important. Writing benchmarks is a very popular and correct way to visualize a software performance. Benchmarks help us to keep performance and confirm effects of optimizations. For keeping the performance, it's key to monitor the benchmark results for changes to the software. To notice performance regression quickly, charts of benchmarking results are useful.

However, there is no good free tool to watch the performance easily and continuously (as far as I looked). So I built a new tool on top of GitHub Action.

This action takes a file which contains benchmark output and updates GitHub pages branch automatically.

Run your benchmarks on your workflow and store the output to a file. tee command is useful to output results to both console and file.

e.g.

- name: Run benchmark
  run: go test -bench 'Benchmark' | tee output.txt

Please add rhysd/github-action-benchmark@{ver} to your workflow yaml file.

e.g.

- name: Store benchmark result
  uses: rhysd/github-action-benchmark@v1
  with:
    name: My Project Go Benchmark
    tool: 'go'
    output-file-path: output.txt

In the example, name, tool, output-file-path inputs are set and other inputs are set to default values. Please read next section to know each input.

Above action step updates your GitHub pages branch automatically. By default it assumes gh-pages branch. Please make it on your remote repository in advance.

Then push the branch to your remote.

As of now, deploying GitHub Pages branch fails with $GITHUB_TOKEN automatically generated for workflows. $GITHUB_TOKEN can push branch to remote, but building GitHub Pages fails. Please read issue #1 for more details.

To avoid this issue for now, you need to create your personal access token.

1. Go to your user settings page
2. Enter 'Developer settings' tab
3. Enter 'Personal access tokens' tab
4. Click 'Generate new token' and enter your favorite token name
5. Check public_repo scope for git push and click 'Generate token' at bottom
6. Go to your repository settings page
7. Enter 'Secrets' tab
8. Create new PERSONAL_GITHUB_TOKEN secret with generated token string

This is a current limitation only for public repositories. For private repository, secrets.GITHUB_TOKEN is available. In the future, this issue would be resolved and we could simply use $GITHUB_TOKEN to deploy GitHub Pages branch. Let's back to workflow YAML file.

There are two options for pushing GitHub pages branch to remote from workflow.

1. Give your API token to github-token input and set auto-push to true
2. Add step for executing git push within workflow

e.g.

- name: Store benchmark result
  uses: rhysd/github-action-benchmark@v1
  with:
    name: My Project Go Benchmark
    tool: 'go'
    output-file-path: output.txt
    github-token: ${{ secrets.PERSONAL_GITHUB_TOKEN }}
    auto-push: true

Just after generating a commit to update benchmark results, github-action-benchmark pushes the commit to GitHub Pages branch. As a bonus, this action pulls the branch before generating a commit. It helps to avoid conflicts among multiple workflows which deploy GitHub pages branch.

e.g.

- name: Push benchmark result
  run: git push 'https://you:${{ secrets.PERSONAL_GITHUB_TOKEN }}@github.com/you/repo-name.git' gh-pages:gh-pages

If you don't set auto-push input, this action does not push changes to remote automatically. This might be an option if you don't want to give API token to this action. Instead, So you need to push GitHub pages branch by your own.

Note that GitHub pages branch and a directory to put benchmark results are customizable by inputs.

After first job execution, https://you.github.io/dev/bench should be available like examples of this repository.

Please read README.md files at each example directory.

• cargo bench for Rust projects
• go test for Go projects
• Benchmark.js for JavaScript/TypeScript projects
• pytest-benchmark for Python projects with pytest

These examples are run in workflows of this repository as described in 'Examples' section above.

Input definitions are written in action.yml.

name and tool must be specified in workflow at uses section of job step.

Other inputs have default values. By default, they assume that GitHub pages is hosted at gh-pages branch and benchmark results are available at https://you.github.io/repo-name/dev/bench.

If you're using docs/ directory of master branch for GitHub pages, please set gh-pages-branch to master and benchmark-data-dir-path to directory under docs like docs/dev/bench.

Please ensure that your benchmark workflow runs only on your branches. Please avoid running it on pull requests. If branch were pushed to GitHub pages branch on pull request, anyone who creates a pull request on your repository could modify your GitHub pages branch.

For this, you can specify branch which runs your benchmark workflow on on: section. Or set proper condition to if: section of step which pushes GitHub pages.

e.g. Runs on only master branch

on:
  push:
    branches:
      - master

e.g. Push when not running for pull request

- name: Push benchmark result
  run: git push ...
  if: github.event_name != 'pull_request'

As far as watching the benchmark results of examples in this repository, amplitude of the benchmarks is about +- 10~20%. If your benchmarks use some resources such as networks or file I/O, the amplitude might be bigger.

If the amplitude is not acceptable, please prepare a stable environment to run benchmarks. GitHub action supports self-hosted runners.

This action creates the default index.html in the directory specified with benchmark-data-dir-path input. By default every benchmark test case has its own chart in the page. Charts are drawn with Chart.js.

If it does not fit to your use case, please modify the HTML file or replace it with your favorite one. Every benchmark data is stored in window.BENCHMARK_DATA so you can create your favorite view.

This action conforms semantic versioning 2.0.

For example, rhysd/github-action-benchmark@v1 means the latest version of 1.x.y. And rhysd/github-action-benchmark@v1.0.2 always uses v1.0.2 even if newer version is published.

master branch of this repository is for development and does not work as action.

• Allow user defined benchmark tool
  • Accept benchmark result as an array of benchmark results as JSON. User can generate JSON file to integrate any benchmarking tool to this action
• Allow to upload results to metrics service such as mackerel instead of updating GitHub pages
• Support pull requests. Instead of updating GitHub pages, add comment to the pull request to explain benchmark result.
• Add an alert comment to commit page when the benchmark result of the commit is far worse than previous one.
• Add more benchmark tools:
  • Google's C++ Benchmark framework
  • airspeed-velocity Python benchmarking tool

the MIT License