UI testing on iOS, without busy waiting
At PSPDFKit, we’ve been using KIF (“Keep It Functional”) since 2014 to test UI components. We wrote about our process in April 2016, and it’s time to revisit. KIF is a good and proven solution. It mostly synthesizes UITouch objects or related events and the app processes it just as if the user would tap. KIF is compiled into your app and can run in tandem with other XCTests. You can mix XCTAssert* calls and UI instructions.
KIF
The big problem is that KIF for us was very flaky. If the system was under high load, tests failed because the timeout wasn’t enough. The solution usually was increasing the timeouts, until we reached a point where UI tests alone were running over 30 minutes. Without compile times.
That’s where we said “no more” and spent a lot of time on fixing the problem. Our solution for “ludicrous speed” was a combination of increasing layer speed and busy waiting. Simply checking the condition every X milliseconds until it’s either true or the timeout hits. This made our tests both faster and more stable. It’s a pragmatic solution.
KIF was originally written by Square and is still pretty well maintained. There are some helpers for Swift available, however the matchers are inconsistent both in scope and naming and some macros are named very general, such as system which causes issues. It’s not a big deal and we have an internal fork that renamed that, but you’ll certainly feel that it’s an older project with a good bit of technical debt. There’ve been talks for fixing up much of that in a KIF 4 and it’s not impossible to improve.
History
Apple released tools for UI automation all the way back in iOS 4. It was called UIAutomation and integrated into… wait for it… Instruments, and you had to use JavaScript to write tests. It all sounded pretty amazing despite the rumors that it was written by an intern. However, it was really flaky and almost no one actually succeeded in building reliable tests on top of it - and the few who did had to build additional tooling around it. UIAutomation was deprecated in Xcode 7 and finally completely removed in Xcode 8, without a clear path to migrate. Moving tests to XCUI was not trivial, but given that almost nobody used UIAutomation, Apple did not bother to write a migrator. Facebook is a notable large exception which used UIAutomation as their WebDriver.
XCUI
With Xcode 7 Apple gave us API to app UI, that can be used from Objective-C and Swift. No more JavaScript. It’s also for iOS and macOS, where UIAutomation was iOS specific. XCUI is quite awesome and even can automatically record tests for you.
Tests in XCUI run in a separate process. This is good in a way that you’re testing closer to what you ship to your users and bad in the way that it makes it very hard to mock data or test model assumptions - which really is how we build almost all of our tests. And if you use workarounds that allow data mocking, you risk polluting your app with test data.
XCUI also didn’t solve the synchronization problem and the documentation recommends using func expectation(for predicate: NSPredicate, evaluatedWith object: Any, handler: XCPredicateExpectationHandler? = nil) -> XCTestExpectation.
The expectation periodically evaluates the predicate and also may use notifications or other events to optimistically re-evaluate.
This is similar to our PSPDFWaitForConditionWithTimeout helper that we use in KIF.
What else is out there?
We’re not talking about Appium which currently has some troubles, because their WebDriver used UIAutomator which has been removed in Xcode 8, so they can’t run tests with Xcode 8 yet, but replacements are in the works).
Other noticable mentions are Subliminal (discontinued) or Frank (abandoned). There’s also Calabash which is backed by Xamarin and open-source. Tests are written with Ruby and it uses a WebDriver (UIAutomator, now custom).
EarlGrey
EarlGrey has been open sourced by Google in February 2016 and you might think it’s quite a young project. However, of you look at the Initial Commit with over 20 KLOC you quickly see that this is not the case. Google has been using EarlGrey internally for years before they made the effort to clean it up and open source it. It’s largely inspired by Android’s Espresso, and that’s a good thing. We’re using Espresso heavily to test our Android SDK, and it’s one of the shining parts of the Android ecosystem. Espresso has some great features such as CSS selector matchers that eventually will trickle down to EarlGrey.
It’s somewhat similar to KIF, however it’s not timeout based but uses extensive synchronization, with means that you dont need explicit waits or sleeps. It’s matchers are much more extensive and consistent and it has better visibility checkers to only interact with elements that really are on the screen - catching bugs that the user might experience.
Enabling Clang’s Address Sanitizer is a known issue in EarlGrey. If you try it, you will get a crash like this one:
However, this is being worked on and it’s quite the read!
Both KIF and EarlGrey extensively use private API to inject touches, which can break on major iOS updates, as it did with iOS 8. One scary looking example is IOHIDEventCreateDigitizerFingerEvent, however even WebKit uses that in it’s test runner, so chances are high that this will keep working.
Demo Time
But don’t trust us - see for yourself! We have an open source project that compares KIF, XCUI and EarlGrey using a demo of PSPDFKit.