Article summary
I’ve been working on developing an iOS app in Swift. It’s my first experience developing in pure Swift, without any Objective-C. This project has taught me a lot about the current state of testing in Swift, including different testing approaches and best practices. In this post, I’ll share some of my experiences and discuss how we have approached testing different types of Swift code. I’ll also talk about some useful testing libraries.
XCTest
XCTest has been the standard out-of-the-box iOS testing framework for as long as I can remember. This is what you get by default in Swift. Though it has been around for a while, I had not used XCTest much in the past. Instead, I usually opted for Kiwi when working in Objective-C. Unfortunately, Kiwi is not supported in Swift. I wanted to give vanilla XCTest a try, so for the first few months, that’s all I used.
On one hand, I learned that XCTest is a very bare-bones and limited testing framework. This wasn’t a particularly surprising revelation—I think most people find it to be average at best.
On the other hand, I also found that you can test most things with a high success rate using just XCTest. The tests may not be the most beautiful, and they may require a lot of boilerplate, but you are usually able to find some way to write the test you want.
General Test Structure
When writing tests in XCTest, you usually create a new class that extends XCTestCase, and add your tests as methods to your new class. It usually looks like this:
class MyClassTests: XCTestCase {
func testCaseA() {
...
}
func testCaseB() {
...
}
}
Synchronous Tests
Synchronous tests are usually straightforward. You instantiate the object you wish to test, call a method, and then use one of the XCTest assertions to confirm the outcome that you expect.
func testAddTwoNumbers {
let adder = MyAdderClass()
let result = adder.add(4, otherNumber: 8)
XCTAssertEqual(result, 12)
}
Asynchronous Tests
Asynchronous tests are a little more tricky, though you can usually use XCTest’s XCTestExpectation class. As an example, suppose that we have a class that takes a number as input, makes a network call to get a second number, adds them together, and calls a callback with the result. To test something like this, we probably want to be able to stub the network call to return a known value, and assert that the result callback contains an expected value. For the sake of clarity, suppose this class looks like this:
class NetworkAdder {
func add(userProvidedNumber: Int, callbackWithSum: (Int) -> ()) {
self.getNumberFromNetwork({ numberFromNetwork in
let sum = userProvidedNumber + numberFromNetwork
callbackWithSum(sum)
})
}
func getNumberFromNetwork(callback: (Int) -> ()) {
let numberFromNetwork = // some operation that get a number
callback(numberFromNetwork)
}
}
The standard way to test this using XCTest would be to extend NetworkAdder
with an inline class, and override getNumberFromNetwork
to return a fixed value. Then you can use some XCTest assertions in the callback you pass into callbackWithSum
. However, you need to ensure that the test waits until the assertions are checked before exiting. To do this, you can use the XCTestExpectation class:
class NetworkAdderTests: XCTestCase {
class MockNetworkAdder: NetworkAdder {
override func getNumberFromNetwork(callback: (Int) -> ()) {
callback(5) // force this to return 5 always for the test
}
}
func testAddAsync() {
let expectation = expectationWithDescription("the add method callback was called with the correct value")
let networkAdder = MockNetworkAdder()
networkAdder.add(8, callbackWithSum: { sum in
XCTAssertEqual(sum, 13)
expectation.fulfill()
})
waitForExpectationsWithTimeout(1, handler: nil)
}
}
While the above mocking strategy requires a lot of boilerplate, it does allow you to test a wide variety of scenarios. In fact, I have found that most scenarios can be tested with some combination of the above synchronous and asynchronous examples.
Testing View Controllers
View controllers are another central testing concern. They can usually be tested effectively using UITests, which I’ll discuss later. However, sometimes unit tests are more appropriate. I have found that if you want to unit test a view controller, it’s important to instantiate it programmatically from your storyboard. This ensures that all of its outlets are properly instantiated as well. I have had several scenarios where I wanted to test the state of one or more view controller outlets at the end of a test (e.g., the text of a UILabel, the number of rows in a table view, etc.).
You can instantiate your view controllers using the storyboard by doing the following:
let mainStoryboard: UIStoryboard = UIStoryboard(name: "Main", bundle: nil)
let myViewController = mainStoryboard.instantiateViewControllerWithIdentifier("MyViewControllerIdentifier") as! MyViewController
myViewController.loadView()
The above code assumes that you have set the identifier on MyViewController
to MyViewControllerIdentifier. I usually run something similar to the above snippet in the before each block for my MyViewController tests.
Upgrading with Quick and Nimble
Although I was able to test most things effectively using XCTest, it didn’t feel great. My tests were often verbose, didn’t read well, and would sometimes require a lot of boilerplate code.
I wanted to try a third-party testing framework to see if it alleviated any of these issues. Quick and Nimble seem to be the defacto third-party Swift testing framework.
Quick is a testing framework that provides a testing structure similar to RSpec or Kiwi. Nimble is a library that provides a large set of matchers to use in place of XCTest’s assertions. Quick and Nimble are usually used together, though they don’t absolutely need to be.
The first thing that you get with Quick and Nimble is much better test readability. The above synchronous test written using Quick and Nimble becomes:
describe("The Adder class") {
it(".add method is able to add two numbers correctly") {
let adder = MyAdderClass()
let result = adder.add(4, otherNumber: 8)
expect(result).to(equal(12))
}
}
Similarly, the asynchronous test becomes:
describe("NetworkAdder") {
it(".add works") {
var result = 0
let networkAdder = MockNetworkAdder()
networkAdder.add(8, callbackWithSum: { sum in
result = sum
})
expect(result).toEventually(equal(13))
}
}
The other really helpful item you get out-of-the-box with Nimble is the ability to expect that things don’t happen in your tests. You can do this via expect(someVariable).toNotEventually(equal(5))
. This makes certain asynchronous tests much easier to write compared to using XCTest, such as confirming that functions are never called, exceptions are never thrown, etc.
Overall, I would strongly recommend using Quick and Nimble over XCTest. The only potential negative that I’ve observed is that XCode seems to get confused more easily when running and reporting results for Quick tests. Sometimes the run button doesn’t immediately appear next to your test code, and sometimes it can forget to report results or even run some tests when running your full test suite. These issues seem to be intermittent and are usually fixed by re-running your test suite. To be fair, I have also observed XCode exhibit the same behavior for XCTests; it just seems to happen less frequently.
UI Testing
The last item I would like to discuss is UITests. In the past, I have used KIF or something similar to write integration-style UI tests.
I initially tried to get KIF working, but experienced some difficulty getting it to build and work in our Swift-only project. As an alternative, I decided to try the UITest functionality built into XCode, and I’m glad that I did. I have found the UI tests to be extremely easy to write, and we have been able to test large amounts of our app using them.
UITests work similarly to KIF or other such test frameworks—they instantiate your application and use accessibility labels on your UI controls to press things in your app and navigate around. You can watch these tests run in the simulator, which is pretty neat. While navigating around, you can assert certain things about your app, such as the text on a UILabel, the number of rows in a UITableView, the text they are displaying, etc.
Let’s walk through an example UITest for an app that contains a button that adds rows to a UITableView, and updates a label with the number of rows in the table. The test will press the button three times and check that a row is added for each press, and the label text is updated appropriately. The app looks like this:
The UITest code looks like this:
func testAddRowsToTable() {
let app = XCUIApplication()
let addRowButton = app.buttons["addRowToTableButton"]
XCTAssertEqual(app.tables["tableView"].cells.count, 0)
XCTAssertEqual(app.staticTexts["numTableViewRowsLabel"].label, "The table contains 0 rows")
addRowButton.tap()
XCTAssertEqual(app.tables["tableView"].cells.count, 1)
XCTAssertEqual(app.staticTexts["numTableViewRowsLabel"].label, "The table contains 1 row")
addRowButton.tap()
XCTAssertEqual(app.tables["tableView"].cells.count, 2)
XCTAssertEqual(app.staticTexts["numTableViewRowsLabel"].label, "The table contains 2 rows")
addRowButton.tap()
XCTAssertEqual(app.tables["tableView"].cells.count, 3)
XCTAssertEqual(app.staticTexts["numTableViewRowsLabel"].label, "The table contains 3 rows")
}
To set this test up correctly, the Add Row To Table button accessibility label was set to addRowToTableButton, the UITableView’s accessibility identifier was set to tableView, and the bottom UILabel’s accessibility identifier was set to numTableViewRowsLabel. You can watch the test run in the simulator—it looks like this:
We found it helpful to create a UITest base class where we could set up an application and do some other configuration work. This class looks like this:
import XCTest
class BaseUITest: XCTestCase {
var app: XCUIApplication?
override func setUp() {
super.setUp()
app = XCUIApplication()
app!.launchArguments.append("UITesting")
continueAfterFailure = false // set to true to continue after failure
app!.launch()
sleep(1) // prevents occasional test failures
}
}
There are a few things to note in the above code sample. The first and most obvious is the sleep(1)
before returning from the setUp
function. I noticed that some of our tests would fail without this—presumably because the test would start running before the app was up and running in the simulator.
Additionally, we are passing a UITesting
string value into our app launch arguments. Occasionally, you will need to mock things out for your UITests (e.g., network calls, file IO, etc.). The best way we found to do this is to set a test-specific launch argument that we can check for in our code and inject UITest classes instead of production classes when injecting our app dependencies on startup. This was mostly inspired by this Stack Overflow post.
Recording Tests
A great, and often overlooked, UITest feature is the ability to record interaction sequences with your app and have XCode write your UITest for you. This won’t add any assertions into your test, but it will create a sequence of UI interactions that you can use as a starting point for your test. You can start a recording by pressing the red record button on the bottom of the screen. This will launch your app and allow you to start using it. Each screen interaction is immediately translated to a line of code that you can see show up dynamically in your test function. When you’re finished, you simply press stop recording.
UI Testing Gotchas
Aside from that one-second delay that we added to the beginning of our UI tests to prevent periodic test failures, there are a few other tricky items to be aware of.
If your test involves typing text into a text field, you need to tap on the text field first to bring it into focus (similar to what you would do if you were using the app). Additionally, you need to ensure that Connect Hardware Keyboard is unchecked on the simulator. This allows the onscreen keyboard to be used when typing into text fields, instead of your laptop keyboard.
When attempting to programmatically access elements from your app, the XCode accessibility UI shows how each control is categorized. You can modify this by selecting different categories. So, for example, to access a UIButton via app.buttons["myButtonIdentifier"]
, your control element needs to be categorized as a Button.
Most of the time, you won’t re-categorize elements, but this screen allows you to look up how to access each control in your app.
Another thing to be careful with is understanding where your app starts when it is launched. If you have an app that requires user login, and it either starts on the login screen if the user is not logged in, or takes the user to the app if they are, your UI tests need to be aware of this. In our tests, we first check to see if the user is logged in and either continue running or log them in/out as desired.
One minor detail in your **Testing View Controllers** section, Apple’s docs say [not to call `loadView` directly and instead access the property `view`](https://developer.apple.com/library/ios/documentation/UIKit/Reference/UIViewController_Class/#//apple_ref/swift/instm/UIViewController/c:objc(cs)UIViewController(im)loadView).
Thanks for the correction. We were calling that to instantiate the outlets on the vc. Does just accessing the view property instantiate them? – looks like it might http://stackoverflow.com/questions/12523198/storyboard-instantiateviewcontrollerwithidentifier-not-setting-iboutlets
yes, it will.
Very nice overview. Thanks for sharing your thoughts.
Which are the advantages of testing like this? It’s more meaningful for big projects, isn’t it?
It doesn’t seem so trivial…
Now I’m testing my functions inside my app and this work as well.
(Sorry for bad english)
I found a slight issue with how your doing your tableViewCell counting. Because TableViews reuse cells, I found that if I was using a filtering system and wanted to check if less cells existed, that I would get back a larger number than expected because tableViews keep their cells even if they are not using them, and then simply recycle them. I was able to solve this by checking for the “hittable” attribute being true on cells, and using that to count visible cells (rather than old ones)
Thanks for the reply Michael – you raise a great point. Using the “hittable” attribute seems like a reasonable thing to do – I’ll keep this in mind in the future.
Hi Matt,
Nice write-up.
Question re: “Additionally, you need to ensure that Connect Hardware Keyboard is unchecked on the simulator. ”
Is there any programmatic way to ensure this is set correctly, mainly for CI testing.
Thanks!
Mike
Great question Mike – see the 2nd answer here – http://stackoverflow.com/questions/32184837/ui-testing-failure-neither-element-nor-any-descendant-has-keyboard-focus-on-se/
I think
defaults write com.apple.iphonesimulator ConnectHardwareKeyboard 0
sholud work in a CI environment.Hi Matt.
Thank you for this usefull tuto.
As you know, it is l trivial to SUM The values of a spreadsheet’s column, but frustrating in Xcode to the point of impossibility. None of the pieces of code i found did the trick. An idea ?
Y U NO post code?
Great post thanks. Were you able to get Quick & Nimble working for your UI tests?