Skip to content

Conversation

@jameslamb
Copy link
Collaborator

@jameslamb jameslamb commented Jul 12, 2024

Release checklist:

Copied from #6439 with a few changes.

deadline

August 3, 2024 (ref: #6522)

before merge

Items to be addressed before merging:

after merge

code to update tags (click me)
NEW_VERSION="4.4.0"
git checkout master
git pull upstream master
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag "v${NEW_VERSION}"
git push upstream stable "v${NEW_VERSION}"

after the release is created by Azure DevOps job

  • Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.
  • Upload release to CRAN
  • Upload release to test PyPI
  • Upload release to PyPI.
code to publish to PyPI (summary)
mkdir /tmp/lgb-release
cd /tmp/lgb-release

NEW_VERSION="4.4.0"

# NOTE: requires a GitHub personal access token with "repo" scope
gh release download \
    --repo microsoft/LightGBM \
    --dir ./artifacts \
    --pattern 'lightgbm*-py3-*.whl' \
    --pattern "lightgbm-${NEW_VERSION}.tar.gz" \
    "v${NEW_VERSION}"

# NOTE: manually downloaded arm64 wheels from CI

# config docs: https://packaging.python.org/en/latest/specifications/pypirc/
twine upload \
    -r testpypi \
    ./artifacts/*

(gh is the GitHub CLI, see https://cli.github.com/manual/gh_release_download)

Then confirmed that installing the latest wheel works.

pip install -i https://test.pypi.org/simple/ "lightgbm==${NEW_VERSION}"
python ./examples/python-guide/logistic_regression.py

Then pushed them to real PyPI.

twine upload \
    ./artifacts/*
code to open that dev version PR (click me)
RELEASE_PR_NUMBER="6439"
RELEASE_VERSION=$(cat ./VERSION.txt)
DEV_VERSION="${RELEASE_VERSION}.99"

git checkout -b ci/dev-version

echo "${DEV_VERSION}" > ./VERSION.txt
sed \
    -i .bak \
    "s|${RELEASE_VERSION}|${DEV_VERSION}|g" \
    .appveyor.yml

sed \
    -i .bak \
    "s|version = \"${RELEASE_VERSION}\"|version = \"${DEV_VERSION}\"|g" \
    ./python-package/pyproject.toml

sed \
    -i .bak \
    's|mode\: release|mode\: unreleased|g' \
    ./R-package/pkgdown/_pkgdown.yml

docker run \
    --rm \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    ubuntu:22.04 \
    ./R-package/recreate-configure.sh

git add \
    ./.appveyor.yml \
    ./R-package/configure \
    ./R-package/pkgdown/_pkgdown.yml \
    ./VERSION.txt \
    ./python-package/pyproject.toml

commit_msg="bump development version to ${DEV_VERSION}"
git commit -m "${commit_msg}"
git push upstream ci/dev-version

gh pr create \
    --repo microsoft/LightGBM \
    --base 'master' \
    --label 'maintenance' \
    --title "${commit_msg}" \
    --body "now that v${RELEASE_VERSION} has been released: #${RELEASE_PR_NUMBER}"

Notes for Reviewers

I believe this should be v4.5.0 instead of v4.4.1 because of some small breaking changes:

See https://github.com/microsoft/LightGBM/releases.

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jul 13, 2024

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/9916742190

Status: success ✔️.

readme = "README.rst"
requires-python = ">=3.7"
version = "4.4.0.99"
version = "4.4.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it was a typo.

Suggested change
version = "4.4.0"
version = "4.5.0"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah!! yes, thank you. Maybe in the future we should have a script that automates these changes, like my colleagues do in the RAPIDS repos: https://github.com/rapidsai/cuml/blob/branch-24.08/ci/release/update-version.sh

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very sorry for resolving this without answering you @borchero , that was unintentional.

I'm -1 on using tools like setuptools-scm or versioneer here. In my experience, they introduce much more complexity than they reduce, at least for projects like LightGBM with a linear sequence of releases (no long-term-support branches, backporting, etc.).

For example, they can be the source of hard-to-debug differences between local and CI builds, unless you're diligent about running git fetch upstream --tags or similar regularly.

You might find some of the other discussion in https://discuss.python.org/t/please-make-package-version-go-away/58501 interesting (although I don't agree with the original post... I believe we should continue to support lightgbm.__version__).

jameslamb and others added 2 commits July 13, 2024 09:50
@StrikerRUS
Copy link
Collaborator

I believe this should be v4.5.0 instead of v4.4.0 because of some small breaking changes:

I guess you meant v4.4.1. Agree with your proposal!

@jameslamb
Copy link
Collaborator Author

Yes I meant "instead of v4.4.1", just edited it. Sorry for all the typos 😅

@StrikerRUS
Copy link
Collaborator

StrikerRUS commented Jul 13, 2024

Seems that Azure Pipelines builds are constantly failing for the master branch.

image

For example, the latest one link: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16597&view=results

Failing jobs are:

  • macOS cpp_tests
[100%] Built target testlightgbm
/Applications/Xcode_13.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/locale:1417:26: runtime error: member call on address 0x612000003ad8 which does not point to an object of type 'std::streambuf'
0x612000003ad8: note: object is of type 'std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >'
 01 00 00 00  30 40 cf 0c 01 00 00 00  30 ba 41 5b f8 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'std::__1::basic_stringbuf<char, std::__1::char_traits<char>, std::__1::allocator<char> >'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /Applications/Xcode_13.1.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/c++/v1/locale:1417:26 in 
/Users/runner/work/1/s/.ci/test.sh: line 63:  8016 Abort trap: 6           ./testlightgbm

##[error]Bash exited with code '1'.
  • Windows bdist
2024-07-12T18:51:57.5021734Z Downloading pydistcheck-0.7.1-py3-none-any.whl (19 kB)
2024-07-12T18:51:58.6643657Z Installing collected packages: pydistcheck
2024-07-12T18:51:58.8103421Z Successfully installed pydistcheck-0.7.1
2024-07-12T18:51:59.8819463Z Traceback (most recent call last):
2024-07-12T18:51:59.8820899Z   File "<frozen runpy>", line 198, in _run_module_as_main
2024-07-12T18:51:59.8821855Z   File "<frozen runpy>", line 88, in _run_code
2024-07-12T18:51:59.8828650Z   File "C:\Miniconda\envs\test-env\Scripts\pydistcheck.exe\__main__.py", line 7, in <module>
2024-07-12T18:51:59.8829712Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1157, in __call__
2024-07-12T18:51:59.8841284Z     return self.main(*args, **kwargs)
2024-07-12T18:51:59.8848274Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8848970Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1078, in main
2024-07-12T18:51:59.8849427Z     rv = self.invoke(ctx)
2024-07-12T18:51:59.8850949Z          ^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8851311Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 1434, in invoke
2024-07-12T18:51:59.8857514Z     return ctx.invoke(self.callback, **ctx.params)
2024-07-12T18:51:59.8858902Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8861213Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\click\core.py", line 783, in invoke
2024-07-12T18:51:59.8863013Z     return __callback(*args, **kwargs)
2024-07-12T18:51:59.8863771Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8865099Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\cli.py", line 256, in check
2024-07-12T18:51:59.8866530Z     errors += this_check(distro_summary=summary)
2024-07-12T18:51:59.8867326Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8868843Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\checks.py", line 64, in __call__
2024-07-12T18:51:59.8870013Z     has_debug_symbols, cmd_str = _file_has_debug_symbols(
2024-07-12T18:51:59.8870823Z                                  ^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8871605Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 73, in _file_has_debug_symbols
2024-07-12T18:51:59.8877657Z     has_debug_symbols, cmd_str = _look_for_debug_symbols(lib_file=file_absolute_path)
2024-07-12T18:51:59.8878095Z                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8879249Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 48, in _look_for_debug_symbols
2024-07-12T18:51:59.8879935Z     stdout = _run_command(args=[*cmd_args, lib_file])
2024-07-12T18:51:59.8880333Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8880723Z   File "C:\Miniconda\envs\test-env\Lib\site-packages\pydistcheck\shared_lib_utils.py", line 16, in _run_command
2024-07-12T18:51:59.8881941Z     stdout = subprocess.run(args, capture_output=True, check=True).stdout
2024-07-12T18:51:59.8882309Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8883036Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 548, in run
2024-07-12T18:51:59.8896405Z     with Popen(*popenargs, **kwargs) as process:
2024-07-12T18:51:59.8897987Z          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8904669Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 1026, in __init__
2024-07-12T18:51:59.8913959Z     self._execute_child(args, executable, preexec_fn, close_fds,
2024-07-12T18:51:59.8915143Z   File "C:\Miniconda\envs\test-env\Lib\subprocess.py", line 1538, in _execute_child
2024-07-12T18:51:59.8929852Z     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2024-07-12T18:51:59.8930741Z                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-07-12T18:51:59.8933401Z ==================== running pydistcheck ====================
2024-07-12T18:51:59.8934510Z OSError: [WinError 122] The data area passed to a system call is too small

Should these fails be a blockers for this release?

@jameslamb
Copy link
Collaborator Author

Add release branch to RTD versions, trigger a new build, check docs

I'll check again once we're done merging changes here.

@jameslamb
Copy link
Collaborator Author

constantly failing on the master branch

Oh wow, yes I hadn't realized that most master builds for the last 2 weeks have failed with exactly those two issues! I just clicked through https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build?definitionId=1&_a=summary&repositoryFilter=1&branchFilter=2%2C2%2C2%2C2%2C2 to see that.

I do think we should try to figure this out for the release, but let's not block releasing the R package on them. If we get to, say, July 30 and have still not fixed it, I think we should at least do a release of the R package to CRAN, to avoid it being archived there.

Could you document these CI failures in a separate issue, and we could discuss further there?

I'm linking this relevant thing I found, that suggests that one issue could be a very long filepath on Windows: sarugaku/shellingham#8. Maybe that could explain the jobs only failing on master... maybe something about that changes the filepaths used by Azure DevOps or something? I'm not sure.

@StrikerRUS
Copy link
Collaborator

Could you document these CI failures in a separate issue, and we could discuss further there?

Sure, done!

If we get to, say, July 30 and have still not fixed it, I think we should at least do a release of the R package to CRAN, to avoid it being archived there.

Absolutely agree!

@jameslamb
Copy link
Collaborator Author

jameslamb commented Jul 22, 2024

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/10033742645

Status: success ✔️.

@jameslamb jameslamb changed the title WIP: release v4.5.0 Jul 22, 2024
@jameslamb jameslamb marked this pull request as ready for review July 22, 2024 02:43
@jameslamb
Copy link
Collaborator Author

I received the following email from CRAN (Prof Brian Ripley) 3 days ago.

subject: CRAN packages with g++ 14.x warnings

[lightgbm was] given a month's notice with a deadline of Aug 3. As the CRAN
summer shutdown starts on Aug 1, please submit a corrected update no
later than July 31.

I believe we should release the current state of this PR as v4.5.0 as soon as possible. I want to allow enough time for CRAN to reject the package and for us to put up another submission prior to their summer shutdown.

I do not think there are any other open PRs or issues that absolutely MUST be addressed in this release.

@guolinke @borchero @jmoralez @shiyu1994 @StrikerRUS could you please review, and approve if you agree?

I'd like to get at least 2 approvals (and at least 1 from either @shiyu1994 or @guolinke ) before releasing.

Copy link
Collaborator

@borchero borchero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go 🚀

@jameslamb jameslamb merged commit 3f7e608 into master Jul 25, 2024
@jameslamb jameslamb deleted the release/v4.5.0 branch July 25, 2024 14:51
@jameslamb
Copy link
Collaborator Author

Ran the following to create the v4.5.0 tag and update the stable tag.

git checkout master
git pull upstream master
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag v4.5.0
git push upstream stable v4.5.0

(NOTE: I alias this repo to upstream and my fork to origin in my git settings)

https://github.com/microsoft/LightGBM/tags

image

That triggered an Azure DevOps build which should create the release automatically: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16722&view=results

This takes around 90 minutes (because of the QEMU CI job).

I'll check back in a few hours and continue the release then.

@jameslamb
Copy link
Collaborator Author

One test job on the release failed, with what looked like a conda network error. Restarted it.

@jameslamb
Copy link
Collaborator Author

Remove the release branch release/v{n}.{n}.{n} from RTD versions

done

Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.

done!

The release is now live: https://github.com/microsoft/LightGBM/releases/tag/v4.5.0

@jameslamb
Copy link
Collaborator Author

Submitted to CRAN with this additional note:

This release fixes the C++20 issues noted in the "gcc-12" checks at https://cran.r-project.org/web/checks/check_results_lightgbm.html. It is being submitted in response to a request from Prof. Brian Ripley to submit before July 31st.

image
@StrikerRUS
Copy link
Collaborator

Homebrew has been automatically updated (and bottles have been built): Homebrew/homebrew-core#178449.

https://formulae.brew.sh/formula/lightgbm#default:

image

@StrikerRUS
Copy link
Collaborator

Can't find ..macosx_..._arm64.whl macOS wheel for M chips in attached artifacts 🤔
Do we build them?
Seems that we did it for 4.4.0:
image

@StrikerRUS
Copy link
Collaborator

Previously failing CRAN gcc check is now OK! 🎉

https://cran.r-project.org/web/checks/check_results_lightgbm.html
image

@jameslamb
Copy link
Collaborator Author

@jameslamb
Copy link
Collaborator Author

Can't find ..macosx_..._arm64.whl macOS wheel for M chips in attached artifacts 🤔

I haven't figured out a good way to make that automatic yet. They're built on GitHub Actions because (at least as of #6391), Azure DevOps didn't have an arm64 runner image.

So they have to be manually added to the release.

They can be found by clicking "🏠 Summary" on a run of the Python-Package workflow, and scrolling to the bottom. Here's the run for v4.5.0: https://github.com/microsoft/LightGBM/actions/runs/10096384964

Screenshot 2024-07-26 at 8 48 18 AM

I just manually downloaded that artifact, unzipped it, and uploaded the file to https://github.com/microsoft/LightGBM/releases/tag/v4.5.0.

@jameslamb
Copy link
Collaborator Author

Upload release to test PyPI

This is done: https://test.pypi.org/project/lightgbm/4.5.0/

And installing it (on my M2 Mac) worked fine.

pip install \
    --no-deps \
    -i https://test.pypi.org/simple/ \
    lightgbm==4.5.0

python -c "import lightgbm"
python examples/python-guide/sklearn_example.py
@StrikerRUS
Copy link
Collaborator

@jameslamb

They're built on GitHub Actions because (at least as of #6391), Azure DevOps didn't have an arm64 runner image.

Ah, I see! Thanks for the explanation!

* Just checked - still no arm64 macOS runner at Azure 😞

@jameslamb
Copy link
Collaborator Author

Upload release to PyPI.

this is done: https://pypi.org/project/lightgbm/4.5.0/#files

@jameslamb
Copy link
Collaborator Author

put up another PR incrementing development version here in GitHub

done: #6575

@StrikerRUS
Copy link
Collaborator

I think we should add one more release TODO item: "update conda-forge version" since @jameslamb is now official lightgbm feedstock's maintainer.

conda-forge/lightgbm-feedstock#60

@jameslamb
Copy link
Collaborator Author

Agreed! I just added it here so we'll remember it when this PR's description is copied into the next release PR.

@jameslamb
Copy link
Collaborator Author

Update cran-comments.md when new release is accepted on CRAN.

put up a PR: #6578

@jameslamb
Copy link
Collaborator Author

Upload release to NuGet

done: https://www.nuget.org/packages/LightGBM/4.5.0

That should be fully published some time in the next few hours.

image
@jameslamb
Copy link
Collaborator Author

With that, I believe this release is complete! Thanks for the help, everyone.

Really great having you back to help @StrikerRUS , the throughput in this project was great over the last month. This release covers just 6 weeks of changes, but included 49 commits from 16 different contributors (v4.4.0...v4.5.0). I'm very happy about that!

@StrikerRUS
Copy link
Collaborator

@jameslamb Thank you very much for your hard work on the release! 💪

@github-actions
Copy link
Contributor

This pull request has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 31, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

5 participants