Writing Tests in Flutter {Part 3} : How to write golden tests in Flutter

Welcome to part 3 of the flutter testing series. By this point you are already familiar with writing unit tests, widget and page tests in flutter. If you have not already, I highly recommend you to go through the earlier parts in the series:

In this part, you will learn about Golden Tests.

What are Golden Tests?

Structurally, golden tests are very similar to widget tests. While widget tests verify the behaviour of widgets, golden tests verify the visual appearance of those widgets. Golden tests generate a screenshot of the widget and compares it against a reference image. If both the images match, the test will pass. These images that are used as a reference are called goldens and thus the tests are named as golden tests.

Similar to widget test, golden tests can be for a small individual widget or for a group of widgets or the entire page as well.

Why write Golden Tests?

If golden tests are very similar to widget tests, why to even write such tests? After all, in a widget test, we do verify that the necessary widgets for a given state are displayed. Let’s understand this with an example.

Consider the following list items. Each list item has a row which contains a text and an icon widget.

Android list elements each row consisting of a text element and an icon widget.

You can verify that the icon is displayed with the following expect statement.

 
expect(find.byIcon(Icons.favorite_border), findsOneWidget);

Now let’s say someone changes the code and switches the position of the icon with the text field.

User interface changes made to an Android list row i.e. icon moved to the left of the text.

Even though this was an unwanted change to the UI, the widget test will still pass. This is because in a widget test, you are verifying the presence of a widget but not its position. A widget test cannot safeguard you again such UI changes. That is where golden tests come into picture.

Starter Project

If you have not already, clone the flutter-testing repo from here. Checkout the golden-test-start branch and open it in the IDE. You should see the following directory structure.

Directory structure of the golden-test-start branch of our flutter-template.

Unlike the previous parts in this series, the golden-test-start branch already has some setup code. Since the test setup part is same as widget test, you have already seen in part 1 and 2.

Golden Test Setup

In part 2, you wrote tests for individual widgets. In this tutorial, you will write golden tests for an entire page. Most of the setup code is already present. The important files are:

fake_search_view_model.dart: This flutter application uses riverpod for state management and this is where the view model is mocked. This part of mocking will vary based on the state management you are using in your application.

 
class FakeSearchViewModel extends SearchViewModel with Mock {
  FakeSearchViewModel(SearchScreenState state) : super(state);

  String _searchTerm = "";

  @override
  String get searchTerm => _searchTerm;

  updateSearchTerm(String newTerm) {
    _searchTerm = newTerm;
  }
}

test_helpers.dart: This file has a function that will do the one time setup required before all the tests in a test file start running. The setup in this function is again very application specific and will vary according to the libraries that your application is using.

 
void baseSetupAll() {
  FlavorConfig.initialize(
    flavor: Flavor.dev,
    values: const FlavorValues(apiBaseUrl: ""),
  );
}

widget_tester_ext.dart: This is the file you created in part 1 to make it easier to load widgets for testing. Similar to the loadWidgets function you added previously, the loadPageForGolden function extracts all the common parent widgets that are required for a page to load.

 
Future loadPageForGolden({
    required Widget page,
    required AutoDisposeStateNotifierProvider viewModelProvider,
    required AutoDisposeStateNotifierProvider fakeViewModelProvider,
  }) async {
    TestWidgetsFlutterBinding.ensureInitialized();
    SharedPreferences.setMockInitialValues({});
    await EasyLocalization.ensureInitialized();

    await pumpWidget(
      EasyLocalization(
        supportedLocales: const [Locale("en", "US")],
        path: "assets/translations",
        fallbackLocale: const Locale("en", "US"),
        assetLoader: const CodegenLoader(),
        child: Builder(builder: (context) {
          return MaterialApp(
            theme: lightTheme,
            debugShowCheckedModeBanner: false,
            darkTheme: darkTheme,
            themeMode: ThemeMode.dark,
            localizationsDelegates: context.localizationDelegates,
            supportedLocales: context.supportedLocales,
            home: ProviderScope(
              overrides: [
                viewModelProvider.overrideWithProvider(fakeViewModelProvider),
              ],
              child: page,
            ),
          );
        }),
      ),
    );
    await pumpAndSettle();
  }

Writing a Golden Test

Open the search_page_golden_test.dart file located in the presentation/goldens directory.

Showing the location of the search_page_golden_test.dart file in the goldens directory.

It has some code already present to replace the mock state management solution. The _loadPageForGolden helper method is setup to load the SearchPage which you will be testing. It also uses the baseSetupAll method form the test helpers.

 
void main() {
  late FakeSearchViewModel fakeSearchViewModel;

  var _fakeSearchViewModelProvider =
      StateNotifierProvider.autoDispose(
          (ref) {
    fakeSearchViewModel = FakeSearchViewModel(SearchScreenState(
      toolbar: UIToolbar(
        title: LocaleKeys.searchPageTitle,
        hasBackButton: true,
      ),
      showLoading: false,
      searchList: List.empty(),
    ));
    return fakeSearchViewModel;
  });

  setUpAll(baseSetupAll);

  setUp(() {});

  tearDown(() {
    resetMocktailState();
  });

  _loadPageForGolden(WidgetTester tester) async {
    await tester.loadPageForGolden(
      page: const SearchPage(),
      viewModelProvider: searchViewModelProvider,
      fakeViewModelProvider: _fakeSearchViewModelProvider,
    );
  }

Add a widget test to verify if the result list is displayed correctly. Use the _loadPageForGolden method to load the page for the test.

 
testWidgets(
  "Given search results are not empty, When non empty search term is present, Then results should be displayed",
  (tester) async {
    // Given
    await _loadPageForGolden(tester);

    // When

    // Then

  },
);

Now since you are testing a page that reacts to changes in data from the state management solution, you need to update the state with the required data. Set the state with the list that you want to display along with setting the loading state to false. setState is a custom function on the view model that will update the state. Use pumpAndSettle to advance the rendering until the list is loaded and any animation is completed.

 
testWidgets(
    "Given search results are not empty, When non empty search term is present, Then results should be displayed",
    (tester) async {
      // Given
      await _loadPageForGolden(tester);

      // When
      fakeSearchViewModel
          .setState((state) => state.copyWith(showLoading: false, searchList: [
        UICity(
          cityId: 1,
          title: "title",
          locationType: "locationType",
          location: "location",
          isFavourite: true,
        ),
        UICity(
          cityId: 2,
          title: "title 2",
          locationType: "locationType 2",
          location: "location 2",
          isFavourite: false,
        ),
      ]));
      await tester.pumpAndSettle();

      // Then

    },
  );

Next add expect statements similar to what you wrote for widget tests.

 
testWidgets(
    "Given search results are not empty, When non empty search term is present, Then results should be displayed",
    (tester) async {
      // Given
      await _loadPageForGolden(tester);

      // When
      fakeSearchViewModel
          .setState((state) => state.copyWith(showLoading: false, searchList: [
        UICity(
          cityId: 1,
          title: "title",
          locationType: "locationType",
          location: "location",
          isFavourite: true,
        ),
        UICity(
          cityId: 2,
          title: "title 2",
          locationType: "locationType 2",
          location: "location 2",
          isFavourite: false,
        ),
      ]));
      await tester.pumpAndSettle();

      // Then
      expect(find.byType(TextField), findsOneWidget);
      expect(find.byType(SearchPageLoadingShimmer), findsNothing);
      expect(find.text(englishUS[LocaleKeys.noResultsFound]), findsNothing);
      expect(find.byType(UICityListItem), findsNWidgets(2));
    },
  );

At this point, you have a normal widget test for the SearchPage. You can run the test and see the following output.

Results of running the widget test for the SearchPage.

To convert this to a golden test, you will add one more expect statement. The expectLater function can wait for a matcher to complete processing. You will use the expectLater to wait for the matchesGoldenFile function to complete its execution. matchesGoldenFile is a function that matches the png images at the given path to the widget that you provide to the expectLater function. Here you are testing the entire page, so you will select the entire page to be compared.

 
testWidgets(
    "Given search results are not empty, When non empty search term is present, Then results should be displayed",
    (tester) async {
      // Given
      ...

      // When
      ...

      // Then
      ...
      await expectLater(find.byType(SearchPage), matchesGoldenFile("search_page_results_golden.png"));
    },
  );

If you run this test now, it will fail with the following error.

Test failure for the golden test added for the Search Page.

You are asking the test to match the SearchPage widget with the file seach_page_results_golden.png, but this file is not yet created.

To generate the golden file, we need to instruct the testing framework to do so. Run the following command to generate the golden file.

 
flutter test --update-goldens

Once the command finishes running, you will see that the golden file is generated.

A directory showing the generated golden file search_page_results_golden.png

It should look something like this. It might look weird because flutter does not render text or icons in a golden test. Also the size is not of a typical phone, but that of a tablet or a desktop. Because of the bigger size the page is rendering a grid view instead of a list. We will look at how to fix all of these issues later in the tutorial.

Try running the tests normally again. You will notice that it passes this time.

Success results of running the search_page_golden_test.dart.

‍

Pardon for the interruption but you won't regret subscribing to LeadReads, for the love of digital products. Read by Top C Execs.

Join here.

‍

Golden test failures

Open the ui_city_list_item.dart file. It has a Row with two children, Text and IconButton.

 
@override
Widget build(BuildContext context) {
  return Padding(
    padding: const EdgeInsets.all(8.0),
    child: Row(
      mainAxisAlignment: MainAxisAlignment.spaceBetween,
      children: [
        Expanded(child: Text(city.title)),
        IconButton(
          icon: Icon(
            city.isFavourite ? Icons.favorite: Icons.favorite_outline,
            color: Theme.of(context).colorScheme.secondary,
          ),
          onPressed: () {
            sink.add(SearchScreenIntent.toggleFavorite(city: city));
          },
        ),
      ],
    ),
  );
}

Swap the position of the children so that the IconButton is first and the Text is second.

 
@override
  Widget build(BuildContext context) {
    return Padding(
      padding: const EdgeInsets.all(8.0),
      child: Row(
        mainAxisAlignment: MainAxisAlignment.spaceBetween,
        children: [
          IconButton(
            icon: Icon(
              city.isFavourite ? Icons.favorite : Icons.favorite_outline,
              color: Theme.of(context).colorScheme.secondary,
            ),
            onPressed: () {
              sink.add(SearchScreenIntent.toggleFavorite(city: city));
            },
          ),
          Expanded(child: Text(city.title)),
        ],
      ),
    );
  }

Run the golden test again. You will see that it fails. In the error message it also indicated the percentage diff between the test widget and the reference image.

It also creates a failure directory in the same folder as the test. It has 4 files, two files that show the diff between the reference and test image, and the test and the master image.

Open the maskedDiff.png file. This image will should you a combined view of both the master and expected images. It also marks the overlapping widgets for easy identification of error.

Preview of maskedDiff.png showing combined view of both master and expected images.

Open the testImage.png file. This image shows the rendering of the UI in the test. You can see the changes in the icon position that you made earlier.

A preview of testImage.png showing how the UI is rendered.

Now you are able to write golden test and check for difference in you UI in case of any failures.

Don’t forget to revert the changes you made to the UI and also delete the failures folder after you are done inspecting the images.

The Golden Toolkit

The above method of writing golden tests has some drawbacks:

The default size is not similar to a mobile device.
It is difficult to test for multiple device sizes at once.
Text and icons are not rendered.

To solve all of these issues, you will use the golden_toolkit. It is a package that will allow us to render text and icons in tests and also allow testing for multiple device sizes at once.

Add the golden_toolkit dependency to the pubspec.yaml file.

 
dev_dependencies:
  golden_toolkit: ^0.13.0

Get the dependencies by running flutter pub get.

Golden Tests on multiple devices

To update the test to use the golden_toolkit, replace the testWidgets function by the testGoldens function form the library.

 
testGoldens(
    "Given search results are not empty, When non empty search term is present, Then results should be displayed",
    (tester) async {
      // Given
      await _loadPageForGolden(tester);

      // When
      ...

      // Then
      ...
			await expectLater(find.byType(SearchPage), matchesGoldenFile("search_page_results_golden.png"));
    },
  );

Next replace the expectLater function call with multiScreenGolden function form the golden toolkit. You need to provide the function with the tester, the name of the images and the list of devices you want to run the test for. These devices could be some default ones the the library gives you or you can create a custom device of your own.

 
testGoldens(
    "Given search results are not empty, When non empty search term is present, Then results should be displayed",
    (tester) async {
      // Given
      await _loadPageForGolden(tester);

      // When
      ...

      // Then
      ...
			await multiScreenGolden(
        tester,
        "search_page_result",
        devices: [
          Device.iphone11,
          Device.phone.copyWith(name: "smallPhone"),
          const Device(
            name: "custom_device",
            devicePixelRatio: 2,
            size: Size(540, 960),
          ),
        ],
      );
    },
  );

Generate the golden files again by running the following command

 
flutter test --update-goldens

You will see a new goldens directory in the test file’s directory. It contains golden images for all the images specified in the test.

Directory showing the generated golden files. These are created after running the command to update the goldens.

Here is what the iPhone 11 image looks like.

Next time you run the test, it will compare results for all the devices that you have listed.

Rendering Font in Golden Test

The final problem is to render fonts in golden test. You can solve that as well using the golden_toolkit.

Simply add the loadAppFonts function at the very beginning of the test.

 
testGoldens(
    "Given search results are not empty, When non empty search term is present, Then results should be displayed",
    (tester) async {
      // Given
      await loadAppFonts();
      await _loadPageForGolden(tester);

      // When
      ...

      // Then
      ...
    },
  );

Update the golden files to create reference images with fonts and icons.

 
flutter test --update-goldens

Open any golden file. You will see that the fonts and icons are now rendered as expected.

Preview of a golden file showing the fonts and icons being rendered.

The golden toolkit is a powerful tool to have in your testing suite. You can make the code more re-usable by extracting the loadAppFonts and multiScreenGolden functions to some common places so that you don’t have to write it in every single test case.

That’s it! You are now capable of writing golden tests for multiple screen sizes and with font rendering.

Where to go from here?

Try to write more test to cover various states or more diverse device sizes. Checkout the golden-test-end branch for the complete code from this tutorial.

Checkout the Flutter Template for more examples of testing and a reusable test setup.

We hope you enjoyed reading these three parts in the series of Writing Tests on Flutter. Want to join the conversation or comment your learnings? Please head over to our twitter on here.

About the Author

A master of Android and Flutter apps, Shounak is busy leading teams of mobile developers by day. A gaming enthusiast, you can always find him in front of a screen looking for a good movie to watch. An avid photographer in his free time, Shounak tries to capture different landscapes at sunset.