Appium and Selenium Grid with multiple Android versions

[cedricroijakkers]

Hi All,

I have a Selenium Grid running with multiple Android emulators, and there are several scripts connecting to the Grid which end up executing on one of the emulators. This is all working fine.

Until recently, I had two emulators with Android 6.0.1 and Appium, and all tests ran fine. Now, however, I have an app that requires Android 7.1.2 (or just 7 to be precise) so I have set up a new emulator on a VM with Android 7.1.2 and installed Appium on this node. I have connected this node to the hub as well, but now stuff is getting mixed up.

The two Android 6.0.1 emulators have the following Appium node config:

{
"capabilities":
[
{
"browserName": "Android",
"version": "6.0.1",
"maxInstances": 1,
"platform": "ANDROID",
"deviceName": "Android"
}
],
"configuration":
{
"cleanUpCycle": 2000,
"timeout": 20,
"url": "http://192.168.101.103:4723/wd/hub",
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 1,
"port": 4723,
"host":"192.168.101.103",
"register": true,
"registerCycle": 5000,
"hub": "http://192.168.101.100:4445",
"hubPort": 4445,
"cleanUpCycle": 2000,
"hubHost": "192.168.101.100"
}
}

And the 7.1.2 instance has the following:

{
"capabilities":
[
{
"browserName": "Android",
"version": "7.1.2",
"maxInstances": 1,
"platform": "ANDROID",
"deviceName": "Android"
}
],
"configuration":
{
"cleanUpCycle": 2000,
"timeout": 20,
"url": "http://192.168.101.107:4723/wd/hub",
"proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy",
"maxSession": 1,
"port": 4723,
"host":"192.168.101.107",
"register": true,
"registerCycle": 5000,
"hub": "http://192.168.101.100:4445",
"hubPort": 4445,
"cleanUpCycle": 2000,
"hubHost": "192.168.101.100"
}
}

I can see the Android device being registered in the hub console, and I can see there is a different version of the OS running on it.

I have a few different tests, with the following desired capabilities (note, this is Python code):

desired_caps['platformName'] = 'ANDROID'
desired_caps['browserName'] = 'Android'
desired_caps['platformVersion'] = '6.0.1'
desired_caps['deviceName'] = 'Android'

And:

desired_caps['platformName'] = 'ANDROID'
desired_caps['browserName'] = 'Android'
desired_caps['platformVersion'] = '7.1.2'
desired_caps['deviceName'] = 'Android'

When the tests run, it works fine sometimes, but also randonly fails. I have two 6.0.1 emulators and one 7.1.2 emulator, and four scripts requiring 6.0.1 and one script requiring 7.1.2. As long as Selenium Grid decides to run the 6.0.1 script on the 6.0.1 emulator (which is a 2 out of 3 chance) it all works fine. Most of the time it does this, but sometimes it does not. I then end up with the following error:

Unable to find an active device or emulator with OS 7.1.2. The following are available: 192.168.101.106:5555 (6.0.1)

Or the other way around:

Unable to find an active device or emulator with OS 6.0.1. The following are available: 192.168.101.108:5555 (7.1.2)

As you can see, while there are emulators available, Selenium Grid still wrongly directs a session requiring platform version X to a server running platform version Y. The next time when the script runs, it can go to the correct emulator, and it works fine.

I'm probably missing something very silly here, but I have tried many different subtle changes in configuration for both Appium and Grid, but it keeps on directing the test to a wrong instance. Can anyone help me in getting this working? In due time, I will be adding iOS checks to this same grid as well, quite possibly with different versions of iOS emulators, so I really would like to have the Grid do what it is supposed to do.

Any help is greatly appreciated. For now I have worked around the issue by directing the 7.1.2 script straight into the 7.1.2 Appium instance, by means of some Apache trickery, but this is of course not ideal...

[cedricroijakkers]

Since I could not find any information in documentation anywhere, I ended up looking in the code for Selenium grid, and finally found out what the problem was.

When starting the Appium server, you can specify a set of capabilities of this device. When running a test, you specify a set of desired capabilities. The trick here, is that these need to match for 100%, or the Grid server will still randomly distribute jobs to any random available node.

In this specific case, I have change my node config to:

"capabilities":
[
{
"browserName": "Android",
"version": "7.1.2",
"maxInstances": 1,
"platform": "ANDROID",
"platformName": "ANDROID",
"deviceName": "Android"
}
],

And:

"capabilities":
[
{
"browserName": "Android",
"version": "6.0.1",
"maxInstances": 1,
"platform": "ANDROID",
"platformName": "ANDROID",
"deviceName": "Android"
}
],

And the desired capabilities in the script to:

desired_caps['browserName'] = 'Android'
desired_caps['platformName'] = 'ANDROID'
desired_caps['version'] = '7.1.2'
desired_caps['deviceName'] = 'Android'
desired_caps['platform'] = 'ANDROID'

And:

desired_caps['browserName'] = 'Android'
desired_caps['platformName'] = 'ANDROID'
desired_caps['version'] = '6.0.1'
desired_caps['deviceName'] = 'Android'
desired_caps['platform'] = 'ANDROID'

As you can see, all the capabilities (except on the node side for the maxInstances; and on the script side for things like newCommandTimeout, appPackage, appActivity, and appWaitActivity) need to match 100% exactly. If there is one capability more on either side, the Grid matcher class will no longer match, and will send the test to a random node instead.

With this setup, I now have the ideal situation where I have two nodes with Android 6, two nodes with Android 7, and when I fire off two tests for Android 6, they will always execute on the first two nodes. When a third test is fired off, and this is for Android 7, it will execute immediately on one of the free Android 7 nodes, but when I fire off a third test and this is Android 6, it will be queued until one of the two Android 6 nodes is free again.

[sunny_sachdeva]

@cedricroijakkers

Have you ever tried running on real android devices?

[olyv]

Hi @cedricroijakkers
are you running each grid node on a separate machine?

I am trying to run two android emulators (two selenium grid nodes on one physical machine) and it’s a mess. Somehow grid hub redirects tests to emulator which is already supposed to be occupied with a running session.

Apart from that, are you working with native app? As far as I remember, desired capability browserName should be empty for native apps