Skip to content

Conversation

@jameslamb
Copy link
Collaborator

This PR addresses #3065 (review), moving all Windows R CI jobs to GitHub Actions.

Why this is important

This should reduce the total runtime of CI for this project. GitHub Actions allows 20 concurrent builds...AppVeyor (which we currently use) allows 1 😬

While testing on my fork, I found that thanks to this concurrency, all 12 jobs for R run in 12-13 minutes!!!

image

As of this PR, we will have four R Windows jobs all in GitHub Actions:

  • R 4.0 + MSYS2 toolchain
  • R 4.0 + Visual Studio 2019
  • R 3.6 + MINGW toolchain
  • R 3.6 + Visual Studio 2017

You can see the full details of software versions for the GitHub Actions environments here.

What I Changed

We didn't move R jobs into GitHub Actions in #3119 because of the issues with Powershell jobs on GitHub Actions (https://github.community/t/powershell-steps-fail-nondeterministically/115496) / Powershell itself (PowerShell/PowerShell#12823).

R, Rscript, and initexmf all produce harmless messages to stderr, which could trigger the issues linked above. I tried many approaches to fix this, including:

  • redirecting to $null
  • setting $ErrorActionPreference = "SilentlyContinue"
  • using -ErrorAction flag
  • piping to Out-Null with |
  • piping to Out-String with |
  • redirecting to a file

None of these approaches worked, but the one in this PR did: calling these commands with R and using R's built-in redirection, sink().

@jameslamb jameslamb requested a review from StrikerRUS June 16, 2020 03:27
@jameslamb jameslamb requested a review from Laurae2 as a code owner June 16, 2020 03:27
Copy link
Collaborator

@StrikerRUS StrikerRUS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jameslamb Wow! Surprised how fast you've got it work! You are truly magician!

Is everything OK with return code from Run-R-Code-Redirect-Stderr?

Just some minor comments:

jameslamb and others added 2 commits June 16, 2020 20:09
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
@StrikerRUS
Copy link
Collaborator

@jameslamb Very strange error at master r-package (windows-2016, MSVC, R 3.6):

2020-06-22T20:39:00.9756578Z Downloading R and Rtools
2020-06-22T20:39:01.0197729Z Downloading https://cloud.r-project.org/bin/windows/base/old/3.6.3/R-3.6.3-win.exe
2020-06-22T20:39:07.7544683Z Downloading https://cloud.r-project.org/bin/windows/Rtools/Rtools35.exe
2020-06-22T20:39:17.8466524Z Installing R
2020-06-22T20:39:54.6747608Z Done installing R
2020-06-22T20:39:54.6748751Z Installing Rtools
2020-06-22T20:40:39.7710780Z Done installing Rtools
2020-06-22T20:40:39.7712023Z Installing dependencies
2020-06-22T20:40:43.0911257Z package 'callr' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1132167Z package 'prettyunits' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1380186Z package 'backports' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1600585Z package 'assertthat' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.1874920Z package 'glue' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2174329Z package 'fansi' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2429247Z package 'desc' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2662949Z package 'pkgbuild' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.2903885Z package 'rprojroot' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.3226199Z package 'rstudioapi' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.3549368Z package 'ps' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.3881218Z package 'cli' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.4292255Z package 'crayon' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.4636846Z package 'digest' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.4783234Z package 'ellipsis' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5258321Z package 'evaluate' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5506307Z package 'magrittr' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5764769Z package 'pkgload' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.5961975Z package 'praise' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.6386148Z package 'rlang' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.6659431Z package 'withr' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.7596015Z package 'data.table' successfully unpacked and MD5 sums checked
2020-06-22T20:40:43.8169609Z package 'jsonlite' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.1616615Z package 'Matrix' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.1981826Z package 'processx' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.2213825Z package 'R6' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.2953141Z package 'testthat' successfully unpacked and MD5 sums checked
2020-06-22T20:40:44.2992830Z 
2020-06-22T20:40:44.2993625Z The downloaded binary packages are in
2020-06-22T20:40:44.2993977Z 	C:\Users\runneradmin\AppData\Local\Temp\RtmpgNi5C6\downloaded_packages
2020-06-22T20:40:44.3519903Z Building R package
2020-06-22T20:40:58.0415255Z ----- build and install logs -----
2020-06-22T20:40:58.0506063Z * checking for file 'D:/a/LightGBM/LightGBM/lightgbm_r/DESCRIPTION' ... OK
2020-06-22T20:40:58.0506283Z 
2020-06-22T20:40:58.0506432Z * preparing 'lightgbm':
2020-06-22T20:40:58.0506821Z 
2020-06-22T20:40:58.0506993Z * checking DESCRIPTION meta-information ... OK
2020-06-22T20:40:58.0507164Z 
2020-06-22T20:40:58.0507639Z * cleaning src
2020-06-22T20:40:58.0507745Z 
2020-06-22T20:40:58.0507902Z * checking for LF line-endings in source and make files and shell scripts
2020-06-22T20:40:58.0507981Z 
2020-06-22T20:40:58.0508163Z * checking for empty or unneeded directories
2020-06-22T20:40:58.0508228Z 
2020-06-22T20:40:58.0508377Z WARNING: directory 'lightgbm/src/compute/test' is empty
2020-06-22T20:40:58.0508510Z 
2020-06-22T20:40:58.0508660Z * looking to see if a 'data/datalist' file should be added
2020-06-22T20:40:58.0508731Z 
2020-06-22T20:40:58.0508877Z * building 'lightgbm_2.3.2.tar.gz'
2020-06-22T20:40:58.0508941Z 
2020-06-22T20:40:58.0509033Z 
2020-06-22T20:40:58.0509084Z 
2020-06-22T20:40:58.0509306Z * installing to library 'd:/a/LightGBM/LightGBM/RLibrary'
2020-06-22T20:40:58.0509418Z 
2020-06-22T20:40:58.0509520Z * installing *source* package 'lightgbm' ...
2020-06-22T20:40:58.0509627Z 
2020-06-22T20:40:58.0509767Z ** using staged installation
2020-06-22T20:40:58.0509827Z 
2020-06-22T20:40:58.0509973Z ** libs
2020-06-22T20:40:58.0510027Z 
2020-06-22T20:40:58.0510218Z       1 [main] make (856) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0xE55410.
2020-06-22T20:40:58.0510430Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0510610Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0512177Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0512409Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0512585Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0512745Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0512948Z       1 [main] make 3824 fork: child -1 - forked process 856 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0513149Z make: d:/a/LightGBM/LightGBM/RLibrary/R/share/make/winshlib.mk:13: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0513364Z       1 [main] make (140) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0xF15410.
2020-06-22T20:40:58.0513595Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0513771Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0513951Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0514247Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0514422Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0514587Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0514783Z   34110 [main] make 3824 fork: child -1 - forked process 140 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0514984Z make: d:/a/LightGBM/LightGBM/RLibrary/R/share/make/winshlib.mk:13: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0515203Z       1 [main] make (520) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0x1085410.
2020-06-22T20:40:58.0515407Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0515540Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0515718Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0515899Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0516074Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0516232Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0516423Z   66502 [main] make 3824 fork: child -1 - forked process 520 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0516623Z make: d:/a/LightGBM/LightGBM/RLibrary/R/share/make/winshlib.mk:13: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0516836Z       1 [main] make (2348) d:\a\LightGBM\LightGBM\RLibrary\Rtools\bin\make.exe: *** fatal error - cygheap base mismatch detected - 0xF35410/0xF15410.
2020-06-22T20:40:58.0517033Z This problem is probably due to using incompatible versions of the cygwin DLL.
2020-06-22T20:40:58.0517205Z Search for cygwin1.dll using the Windows Start->Find/Search facility
2020-06-22T20:40:58.0517341Z and delete all but the most recent version.  The most recent version *should*
2020-06-22T20:40:58.0517515Z reside in x:\cygwin\bin, where 'x' is the drive on which you have
2020-06-22T20:40:58.0517684Z installed the cygwin distribution.  Rebooting is also suggested if you
2020-06-22T20:40:58.0517841Z are unable to find another cygwin DLL.
2020-06-22T20:40:58.0518030Z   98870 [main] make 3824 fork: child -1 - forked process 2348 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
2020-06-22T20:40:58.0518203Z make: fork: Resource temporarily unavailable
2020-06-22T20:40:58.0518357Z ERROR: compilation failed for package 'lightgbm'
2020-06-22T20:40:58.0518427Z 
2020-06-22T20:40:58.0518578Z * removing 'd:/a/LightGBM/LightGBM/RLibrary/lightgbm'
2020-06-22T20:40:58.0518683Z 
2020-06-22T20:40:58.0518894Z ----- end of build and install logs -----
2020-06-22T20:40:58.0610797Z The wrong compiler was used. Check the build logs.
2020-06-22T20:40:58.1127926Z ##[error]Process completed with exit code -1.
@jameslamb
Copy link
Collaborator Author

😱 I have never seen that one!

@github-actions
Copy link
Contributor

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

3 participants