Vapor and Swift Testing: Running tests in parallel

Swift on the Server and Vapor go hand in hand, but using Swift Testing can be challenging initially. Tests run in parallel by default, and it’s likely that your tests are running against a localhost database, which requires serial execution of tests.

Running tests in parallel drastically improves performance. The performance gains are significant but even more impressive if you compare them to hitting an actual database with several required migrations that are often needed in a MySQL project. I’m building a backend using Vapor for RocketSim for Teams and recently switched to Swift Testing. It’s time to dive into the learnings from that migration.

Swift Testing and parallel execution

It’s essential to understand the difference between XCTests and Swift Testing. At first, you might assume you can simply rewrite your Vapor tests to use Swift Testing, but you’ll quickly notice tests are failing. This is due to the nature of running tests in parallel when using Swift Testing.

Most of your Vapor code probably relies on database queries, and you’ll need a database connection to execute these queries during a unit test. However, if multiple tests start to hit that database at the same time, you’ll likely run into flaky tests. This is even more so if you’ve done several migrations over time.

Therefore, you can’t simply rewrite to use Swift Testing and you need to think about running tests in parallel. If you want a quick solution, you can add the following @Suite attribute to all your test classes:

@Suite(.serialized)
final class VerifyDeviceTests {
    /// ... 
}

This will run your existing tests serially, likely resulting in tests that succeed again. I recommend doing this at first while migrating to Swift Testing to verify your refactor, but eventually, you want to migrate to run tests in parallel since it can drastically improve your productivity and workflow.

Using the Repository Pattern to prevent database queries

You can’t run tests in parallel because your tests depend on an actual database connection. The logical solution would be to remove those database queries while running tests and to create an isolated environment for each test. However, you can’t skip testing your database queries, as these are the core of your application. Therefore, we’ll work toward a solution that allows you to run tests using both in-memory and a database connection.

We already have a working codebase that uses database queries, so we must introduce another implementation to query data from an in-memory store. For this, we’ll use the repository pattern, which I explained in detail in my article ‘Repository design pattern in Swift explained using code examples‘.

We start by defining a DatabaseRepository:

protocol DatabaseRepository: Repository {
    var database: Database { get }
    init(database: Database)
}

Followed by defining an actual repository. In this case, one to interact with users in our database:

protocol UserRepository: Repository {
    func create(_ user: MySQLUser) async throws
}

struct DatabaseUserRepository: UserRepository, DatabaseRepository {
    let database: Database
    
    func create(_ user: MySQLUser) async throws {
        return try await user.create(on: database)
    }
}

For tests, we’ll define another repository. In this case, we’ll use an in-memory backing store:

class TestUserRepository: UserRepository, TestRepository {
    
    private var users: [MySQLUser] = []
    
    func create(_ user: MySQLUser) async throws {
        user.id = UUID()
        users.append(user)
    }
}

Note that we’re generating an id ourselves since we can’t rely on the database for this. This is also an excellent point to realize we will maintain two ways of querying data. You need to decide whether you’re up for that, but I decided to do this since regular development will be much faster. In the end, we’ll also have a way to query the actual database, which makes me convinced that both in-memory and database connections are set up correctly.

Creating an application accessor for repositories

After setting up both repositories, we need a way to configure the repository for our application. We will build this using a few layers, starting with a repository provider:

struct RepositoryProvider {
    
    /// A convenience accessor to a database repositories provider.
    static var database: RepositoryProvider {
        RepositoryProvider(configure: {
            $0.repositories.use { DatabaseUserRepository(database: $0.db) }
        })
    }
    
    fileprivate let configure: (Application) -> ()
}

Note that we’re defining these layers in a single file, allowing us to use fileprivate.

Next, we’re creating a Repositories structure and accessor on the Vapor Application instance:

extension Application {
    
    /// An accessor to access the repositories struct from a Vapor application.
    var repositories: Repositories {
        Repositories(app: self)
    }
}

/// Create a struct to access all configured repositories.
struct Repositories {
    private struct RepositoriesStorageKey: StorageKey {
        typealias Value = RepositoryStorage
    }
    
    let app: Application
    
    var storage: RepositoryStorage {
        if app.storage[RepositoriesStorageKey.self] == nil {
            app.storage[RepositoriesStorageKey.self] = .init()
        }
        
        return app.storage[RepositoriesStorageKey.self]!
    }
    
    func use(_ provider: RepositoryProvider) {
        provider.configure(app)
    }
}

The storage makes use of a new RepositoryStorage type:

/// The storage of repository builders that we store inside a Vapor `Application` storage.
final class RepositoryStorage {
    var makeUserRepository: ((Application) -> UserRepository)?
    fileprivate init() { }
}

These layers allow us to configure the default database configuration using the static RepositoryProvider.database property. You can call the configuration at the launch path of your Vapor application:

app.repositories.use(.database)

Making use of our in-memory repository for tests

With all layers in place, we can start configuring unit tests to use the in-memory repositories. The RepositoryStorage is a class that is accessible via the Application storage. Therefore, we can overwrite the default repository as follows:

final class ApplicationTestWorld {
    private let userRepository = TestUserRepository()
    
    init(app: Application) {
        app.repositories.use { _ in self.userRepository }
    }
}

We can then start using this test world in our actual test classes:

struct UserTests {
    let app: Application
    let testWorld: ApplicationTestWorld
    
    init() async throws {
        self.app = try await Application.make(.testing)
        self.testWorld = ApplicationTestWorld(app: app)
    }
    
    /// Write your tests
}

All our tests will use the test repository instead of accessing the database.

Rewriting our implementations to make use of the repository

After configuring all, we need to make sure our implementation accesses user data via the repository only. Since we’ve created an Application accessor, we can access the repository as follows:

request.application.repositories.users.create(MySQLUser(...))

It’s important to review all your code and rewrite it to use the repository layers instead. This is great in general since you centralize all data accessing inside repositories, making it easier to maintain data queries.

Combining both database and in-memory tests

There might be cases in which you always want to run a test on the actual database. For example, you can write a test for the DatabaseUserRepository. In these cases, you can simply rely on your default application configuration and skip overwriting the repositories with an in-memory variant.

However, I decided I didn’t want to write tests for each repository on top of actual implementation logic that already verifies its implementation. Therefore, there’s another solution in which you’ll run all tests on a database.

Using a separate scheme to run all tests on a database

Don’t get me wrong: I started writing this article to run tests in memory. However, I only did so since I wanted to have the option to quickly verify my implementation logic while working on new implementations. Before every release, however, I want to have the option to test all logic on an actual database to ensure things are working as expected for production. Therefore, I’ve set everything up so that we can use a separate scheme to run all the tests with our database repositories.

It starts by defining a computed property on Application:

extension Application {

    /// Some test plans force running on MySQL for integration testing.
    /// This is enabled by passing in the `--force-running-on-mysql` launch argument.
    var forceRunningOnMySQL: Bool {
        environment.arguments.contains("--force-running-on-mysql")
    }
}

The next step is to create a new test plan and configure it with the launch argument. You’ll also need to overwrite the parallel execution setting since we want to run tests serially. For Swift Testing, we can use the --no-parallel argument. Once you’ve created the test plan, you’ll only need to connect it to a new scheme to simplify running tests on an actual database:

Using Swift Testing with Vapor while running tests serially using a custom test plan.

Finally, we can update our test world code to only overwrite with an in-memory repository if the launch argument isn’t set:

final class ApplicationTestWorld {
    private let userRepository = TestUserRepository()
    
    init(app: Application) {
        /// We only register test repositories if we're not running on an actual MySQL database.
        if !app.forceRunningOnMySQL {
            app.repositories.use { _ in self.userRepository }
        }
    }
}

The final implementation lets you switch between two schemes while developing your Swift Vapor backend. Ideally, you’ll run the database scheme on your pull requests to ensure your application is always tested with a database connection.

Conclusion

Swift Testing is great and allows you to write tests using modern Swift APIs. Vapor applications can be tested, too, but require you to ideally write a few extra layers to run tests using an in-memory store during development. Using a custom test plan and scheme, you can switch between running tests on a database vs. the in-memory alternative.

Here are a few more articles for you to make the most out of Swift Testing:

Thanks!

SwiftLee

A weekly blog about Swift, iOS and Xcode Tips and Tricks