Pixel Tests compare one or more screenshots with already-approved images using Skia Gold. They guarantee that the UI does not change its appearance unexpectedly and are a good addition to a portfolio of regression tests.
There are two ways that pixel tests can be written:
Screenshot test verbs (preferred)TestBrowserUi API (provided for legacy support; do not use)All pixel tests must be placed in pixel_tests.filter.
Tests in browser_tests will also be run in the pixel_browser_tests job on eligible builders. Tests in interactive_ui_tests will also be run in the pixel_interactive_ui_tests job on eligible builders.
All methods of taking screenshots allow you to set a baseline CL. This ensures that old expected images are thrown away when the UI is modified. If the baseline were not set, a regression that caused some new UI to appear might pass a pixel test because it matched an older version of the UI surface.
The procedure for baseline CLs is:
To run a pixel test locally, use:
test_executable --gtest_filter=Test.Name --browser-ui-tests-verify-pixels --enable-pixel-output-in-tests --test-launcher-retry-limit=0
Where <test_executable> is either browser_tests or interactive_ui_tests, and <Test.Name> is the full name of your test, with dot.
If you want to see UI as it's being screenshot, replace --enable-pixel-output-in-tests with --test-launcher-interactive - this will freeze the test just after the screenshot is taken. Dismiss the UI to continue the test.
Failed pixel tests will have a URL link to Skia Gold in the test output through which you can view the expected and actual screenshot images. Log in and either accept or reject images that do not match the baseline. Once you‘ve accepted a new image, future tests that produce the same output won’t fail.
Kombucha tests use a declarative syntax to perform interaction testing on the browser. They are the preferred way to create end-to-end interaction and critical user journey regression tests for Chrome Desktop.
Nearly all Kombucha tests can derive directly from InteractiveBrowserTest. InteractiveBrowserTest is a strict superset of InProcessBrowserTest so it is usually safe to simply swap one for the other.
To use Kombucha to pixel-test UI, use the Screenshot or ScreenshotSurface verb. Because these tests will also be run in non-pixel-test mode, you will need to precede the first screenshot with SetOnIncompatibleAction() with an option other than kFailTest (which is the default).
The Screenshot verb takes a picture of exactly the UI element specified.
The ScreenshotSurface verb takes a picture of the entire dialog or window containing the element. Be careful not to capture other elements that are likely to change on their own!
Example:
// Inherit from InteractiveBrowserTest[Api]: class MyNewDialogUiTest : public InteractiveBrowserTest { ... }; // Baseline Gerrit CL number of the most recent CL that modified the UI. constexpr char kScreenshotBaselineCL[] = "12345678"; // Screenshot the feature's entrypoint button, then click it, wait for the // feature's dialog, then screenshot the entire dialog. IN_PROC_BROWSER_TEST_F(MyNewDialogUiTest, OpenAndVerifyContents) { RunTestSequence( SetOnIncompatibleAction( OnIncompatibleAction::kIgnoreAndContinue "Screenshots not supported in all testing environments."), // Grab a screenshot of the toolbar button that is the entry point for the feature. Screenshot(kMyNewToolbarButtonElementId, /*screenshot_name=*/"entry_point", /*baseline_cl=*/kScreenshotBaselineCL) PressButton(kMyNewToolbarButtonElementId), WaitForShow(MyNewDialog::kDialogElementId), // Grab a screenshot of the entire dialog that pops up. ScreenshotSurface(MyNewDialog::kDialogElementId, /*screenshot_name=*/"whole_dialog", /*baseline_cl=*/kScreenshotBaselineCL)); }
Note that a test can take multiple screenshots; they must have unique names. In the above example, entry_point and whole_dialog will be treated as two separate screenshots to be compared against separate Skia Gold masters.
UiBrowserTest and DialogBrowserTest provide an alternate (and older) method of capturing a surface for pixel testing. These are also base classes your test harness needs to inherit from, and also replace InProcessBrowserTest.
For example, assume the existing InProcessBrowserTest is in foo_browsertest.cc:
class FooUiTest : public InProcessBrowserTest { ...
Change this to inherit from DialogBrowserTest (for dialogs) or UiBrowserTest (for non-dialogs), and override ShowUi(std::string). For non-dialogs, also override VerifyUi() and WaitForUserDismissal(). See Advanced Usage for details.
class FooUiTest : public UiBrowserTest { public: .. // UiBrowserTest: void ShowUi(const std::string& name) override { /* Show Ui attached to browser() and leave it open. */ } // These next two are not necessary if subclassing DialogBrowserTest. bool VerifyUi() override { /* Return true if the UI was successfully shown. */ } void WaitForUserDismissal() override { /* Block until the UI has been dismissed. */ } .. };
Finally, add test invocations using the usual GTest macros, in foo_browsertest.cc:
IN_PROC_BROWSER_TEST_F(FooUiTest, InvokeUi_default) { ShowAndVerifyUi(); }
Notes:
ShowAndVerifyUi();”.default” is the std::string passed to ShowUi() and can be customized. See Testing additional UI “styles”.default (in this case) must always be “InvokeUi_”.List the available TestBrowserUi tests:
$ ./browser_tests --gtest_filter=BrowserUiTest.Invoke $ ./interactive_ui_tests --gtest_filter=BrowserInteractiveUiTest.Invoke
E.g. FooUiTest.InvokeUi_default should be listed. To show the UI interactively, run
# If FooUiTest is a browser test. $ ./browser_tests --gtest_filter=BrowserUiTest.Invoke \ --test-launcher-interactive --ui=FooUiTest.InvokeUi_default # If FooUiTest is an interactive UI test. $ ./interactive_ui_tests --gtest_filter=BrowserInteractiveUiTest.Invoke \ --test-launcher-interactive --ui=FooUiTest.InvokeUi_default
BrowserUiTest.Invoke searches for gtests that have “InvokeUi_” in their names, so they can be collected in a list. Providing a --ui argument will invoke that test case in a subprocess. Including --test-launcher-interactive will set up an environment for that subprocess that allows interactivity, e.g., to take screenshots. The test ends once the UI is dismissed.
The FooUiTest.InvokeUi_default test case will still be run in the usual browser_tests test suite. Ensure it passes, and isn’t flaky. This will give your UI some regression test coverage. ShowAndVerifyUi() checks to ensure UI is actually created when it invokes ShowUi("default").
BrowserInteractiveUiTest is the equivalent of BrowserUiTest for interactive_ui_tests.
This is also run in browser_tests but, when run that way, the test case just lists the registered test harnesses (it does not iterate over them). A subprocess is never created unless --ui is passed on the command line.
If your test harness inherits from a descendant of InProcessBrowserTest (one example: ExtensionBrowserTest) then the SupportsTestUi<> and SupportsTestDialog templates are provided. E.g.
class ExtensionInstallDialogViewTestBase : public ExtensionBrowserTest { ...
becomes
class ExtensionInstallDialogViewTestBase : public SupportsTestDialog<ExtensionBrowserTest> { ...
If you need to do any setup before ShowUi() is called, or any teardown in the non-interactive case, you can override the PreShow() and `DismissUi() methods.
Add additional test cases, with a different string after “InvokeUi_”. Example:
IN_PROC_BROWSER_TEST_F(CardUnmaskViewBrowserTest, InvokeUi_expired) { ShowAndVerifyUi(); } IN_PROC_BROWSER_TEST_F(CardUnmaskViewBrowserTest, InvokeUi_valid) { ShowAndVerifyUi(); }
The strings “expired” or “valid” will be given as arguments to ShowUi(std::string).
Bug reference: Issue 654151.
Chrome has a lot of browser UI; often for obscure use-cases and often hard to invoke. It has traditionally been difficult to be systematic while checking UI for possible regressions. For example, to investigate changes to shared layout parameters which are testable only with visual inspection.
For Chrome UI review, screenshots need to be taken. Iterating over all the “styles” that UI may appear with is fiddly. E.g. a login or particular web server setup may be required. It’s important to provide a consistent “look” for UI review (e.g. same test data, same browser size, anchoring position, etc.).
Some UI lacks tests. Some UI has zero coverage on the bots. UI elements can have tricky lifetimes and common mistakes are repeated. TestBrowserUi runs simple “Show UI” regression tests and can be extended to do more.
Even discovering the full set of UI present for each platform in Chrome is difficult.
browser_tests already provides a browser()->window() of a consistent size that can be used as a dialog anchor and to take screenshots for UI review.
Some UI already has a test harness with appropriate setup (e.g. test data) running in browser_tests.
BrowserUiTest should require minimal setup and minimal ongoing maintenance.An alternative is to maintain a working end-to-end build target executable to do this, but this has additional costs (and is hard).
InitializeGLOneOffPlatform(), etc.).Why not chrome.exe?
Opt in more UI!
Automatically generate screenshots (for each platform, in various languages)
(maybe) Try removing the subprocess
Find obscure workflows for invoking UI that has no test coverage and causes crashes (e.g. http://crrev.com/426302)
Find memory leaks, e.g. http://crrev.com/432320
$ ./out/gn_Debug/browser_tests --gtest_filter=BrowserUiTest.Invoke
Note: Google Test filter = BrowserUiTest.Invoke
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from BrowserUiTest
[ RUN ] BrowserUiTest.Invoke
[26879:775:0207/134949.118352:30434675...:INFO:browser_ui_browsertest.cc(46)
Pass one of the following after --ui=
AppInfoDialogBrowserTest.InvokeUi_default
AskGoogleForSuggestionsDialogTest.DISABLED_InvokeUi_default
BluetoothChooserBrowserTest.InvokeUi_ConnectedBubble
BluetoothChooserBrowserTest.InvokeUi_ConnectedModal
/* and many more */
[ OK ] BrowserUiTest.Invoke (0 ms)
[----------] 1 test from BrowserUiTest (0 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1 ms total)
[ PASSED ] 1 test.
[1/1] BrowserUiTest.Invoke (334 ms)
SUCCESS: all tests passed.
$ ./out/gn_Debug/browser_tests --gtest_filter=BrowserUiTest.Invoke --ui=CardUnmaskPromptViewBrowserTest.InvokeUi_expired
Note: Google Test filter = BrowserUiTest.Invoke [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from BrowserUiTest [ RUN ] BrowserUiTest.Invoke Note: Google Test filter = CardUnmaskPromptViewBrowserTest.InvokeDefault [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from CardUnmaskPromptViewBrowserTest, where TypeParam = [ RUN ] CardUnmaskPromptViewBrowserTest.InvokeUi_expired /* 7 lines of uninteresting log spam */ [ OK ] CardUnmaskPromptViewBrowserTest.InvokeUi_expired (1324 ms) [----------] 1 test from CardUnmaskPromptViewBrowserTest (1324 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1325 ms total) [ PASSED ] 1 test. [ OK ] BrowserUiTest.Invoke (1642 ms) [----------] 1 test from BrowserUiTest (1642 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test case ran. (1642 ms total) [ PASSED ] 1 test. [1/1] BrowserUiTest.Invoke (2111 ms) SUCCESS: all tests passed.
$ ./out/gn_Debug/browser_tests --gtest_filter=BrowserUiTest.Invoke --ui=CardUnmaskPromptViewBrowserTest.InvokeUi_expired --test-launcher-interactive
/* * Output as above, except the test are not interleaved, and the browser window * should remain open until the UI is dismissed */