Chromium Docs

Web Apps - Testing

Please read Testing In Chromium for general guidance on writing tests in chromium.

The following tests are expected for writing code in this system:

  • Unit tests
  • Browser tests
  • Integration tests

Known Issues

  • Unit tests currently cannot rely on WebContents functionality, as that is not built as part of unit test frameworks. Instead, they must use the FakeWebAppUrlLoader or FakeWebAppDataRetriever classes.
    • Note: This should be fixed in early 2023. See bug to make it easier to install apps in unit tests that require a web contents and bug to improve the WebContents dependency and current helper classes to allow the WebAppProvider system to not directly depend on WebContents.
  • Installing web apps before the WebAppProvider system starts can be cumbersome.
    • Browser tests can use the PRE_ test functionality to set up any state.
    • Unit tests must either load a static profile directory saved in the test data, or create a test-only way to explicitly delay the desired subsystem from starting.

Unit tests

Unit tests have the following benefits:

  • They are very efficient.
  • They run on all relevant CQ trybots.
  • They will always be supported by the code coverage framework.

Unit tests are the fastest tests to execute and are expected to be used to test most cases, especially error cases. They are usually built on the WebAppTest base class, and use the FakeWebAppProvider to customize (or not) the dependencies of the WebAppProvider system.

Notes

  • UI elements do not work in unit tests, and the appropriate fakes must be used (see External Dependencies).
  • If one of the external dependencies of the system cannot be faked out yet or the feature is tightly coupled to this, then it might make sense to use a browser test instead (or make that dependency fake-able).
  • Please use the WebAppTest base class if possible.
  • Unit tests based on WebAppTest print a snapshot of chrome://web-app-internals to console on test failures. This can be a powerful debugging tool. The command line flag --disable-web-app-internals-log can be used to disable this feature.

Browser tests

With improved web app test support, most of the components should using unittests to cover the detailed test cases.

Creating an integration test (using the integration framework) should satisfy the need for end-to-end tests for major use-cases of your feature. However, you may need to create one due to:

  • The unittest framework doesn’t support certain needs.
  • You need end-to-end test, but using integration test framework has too much overhead.

Browser tests are much more expensive to run, as they run a fully functional browser. These tests are usually only created to test functionality that requires multiple parts of the system to be running or dependencies like the Sync service to be fully running and functional. It is good practice to have browsertests be as true-to-user-action as possible, to make sure that as much of our stack is exercised.

An example set of browser tests are in web_app_browsertest.cc. Please use the WebAppBrowserTestBase base class.

Notes

  • Browser tests based on WebAppBrowserTestBase print a snapshot of chrome://web-app-internals to console on test failures. This can be a powerful debugging tool. The command line flag --disable-web-app-internals-log can be used to disable this feature.

Integration tests

We have a custom integration testing framework that we use due to the complexity of our use-cases. See integration-testing-framework.md for more information.

It is a good idea to think about your integration tests early & figure out your CUJs with the team. Having your CUJs and integration tests working early greatly speeds up development & launch time.

Notes

  • Integration tests using WebAppIntegrationTestDriver print a snapshot of chrome://web-app-internals to console on test failures. This can be a powerful debugging tool. The command line flag --disable-web-app-internals-log can be used to disable this feature.

Testing OS integration

It is very common to test OS integration. By default, OS integration is suppressed if the test extends WebAppTest or WebAppBrowserTestBase.

End-to-end OS integration testing is facilitated using the OsIntegrationTestOverride. If OS integration CAN be tested in an automated way, this class will do so. If not, the existence of this override will stub-out the OS integration at the lowest level to test as much of our code as possible.

Fake* classes

A class that starts with Fake is meant to completely replace a component of the system. They inherit from a base class with virtual methods, and allow a test to specify custom behavior or checks. The component should seem to be working correctly to other system components, but with behavior that is defined by a test.

The most common pattern here is that the Fake will by default appear to work correctly, and a test can either specify it to return custom results, fail in specific ways, or simply check that it was used in the correct way.

An example is fake_os_integration_manager.h, which pretends to successfully perform install, update, and uninstall operations on OS integration, but instead pretends to work and does simple bookkeeping for tests to check that it was called correctly.

Mock* classes

A class that start with Mock is a gmock version of the class. This allows the user to have complete control of exactly what that class does, verify it is called exactly as expected, etc. These tend to be much more powerful to use than a Fake, as you can easily specify every possible case you might want to check, like which arguments are called and the exact calling order of multiple functions, even across multiple mocks. The downsides are:

  • Mocks end up being very verbose to use, often at the expense of test readiability.
  • Mocks require creating a mock class & learning how to use gmock.

These are generally not preferred to a “Fake”.

Tool: FakeWebAppProvider

The FakeWebAppProvider is basically a fake version of the WebAppProvider system, that uses the WebAppProvider root class to set up subsystems and can be used to selectively set fake subsystems or shut them down on a per-demand basis to test system shutdown use-cases.

By default, the FakeWebAppProvider will NOT start the WebAppProvider system, and it must be manually done so. This is usually done by calling AwaitStartWebAppProviderAndSubsystems.

Common issue: Waiting

Many operations that happen at higher levels that the commands / scheduling system in the WebAppProvider require that tests wait for async operations to complete.

Tabs & Browsers

  • AllBrowserTabAddedWaiter - Waits for a tab to be added anywhere (works for both app browser and regular browser).
  • BrowserChangeObserver - Waits for a browser to add or remove.

Navigation & Loading

  • UrlLoadObserver - Waits for given url to load anywhere.
  • content::TestNavigationObserver - Waits for a navigation anywhere or in given WebContents. See StartWatchingNewWebContents to watch all web contents.
  • content::WebContentsObserver - Can generally be used to wait for events on a given content::WebContents
    • To wait for onload to complete in a page, the ::DocumentOnLoadCompletedInPrimaryMainFrame can be used if WebContents::IsDocumentOnLoadCompletedInPrimaryMainFrame() returns false.

WebAppProvider commands

WebAppCommandManager::AwaitAllCommandsCompleteForTesting will wait for all commands to complete. This will mostly handle all tasks in the WebAppProvider.

Common issue: External Dependency that isn't faked

Sometimes classes use a dependency that either doesn‘t work or isn’t fake-able in our system.

  1. Can you just not depend on that? The best way is to remove the dependency entirely if possible.
  2. If there is a way to easily fake the dependency that is already supported, then do that next.
    • e.g. if it's a KeyedService, and the authors have a fake version you can use, then use that. See how it is used elsewhere.
  3. Create a new interface for this new external dependency, put it on the WebAppProvider, and create a fake for it so that you can test with it faked.
  4. If all else fails, use a browser test.