How to generate scripts automatically for Mobile

[Kirthi]

Hi All,

I am working on a POC where we are trying to generate scripts automatically when the user performs action on Mobile devices (Web & Native app on Android & iOS) by connecting it to the system.

We did a similar solution for Desktop web application using JavaFx which captures the actions [click, sendkeys, etc] and the web elements. If the user provides the url and performs the test steps manually once, it will identify the objects and actions and generate selenium scripts accordingly. Now we wanna do the same for Mobile.

Is there any Appium API to do this ??

Please help me out if any one knows how to leverage this.

[simongilmurray]

If you download the Appium desktop client, you can pass in your parameters for App, device etc. and then you can use it to view the element tree and also record macros through it.

[Kirthi]

Thanks for your suggestion @simongilmurray.

My requirement is i’l provide a UI in which use can select the application and click on ‘Start’ button and perform test steps on it. Once the user clicks ‘Stop’ button, script should get generated automatically.

[Pavan_N]

@Kirthi you can do it for java, javascript ,ruby and python in Appium client as said by @simongilmurray is totally agreeable .try to use it and copy to clipboard and past the code in you test class file

[Kirthi]

@simongilmurray & @Pavan_N - I guess you both are referring to Appium Inspector with recorder.

My requirement is I wanna built a new tool similar to that. Does Appium provides any library to do so?

[simongilmurray]

appium/appium-desktop

Go here, download the Appium Desktop client and read through how to set it up.

[Kirthi]

My requirement is I wanna build a new tool similar to that where I can record and generate scripts. Does Appium provides any library that i leverage it?

[Raymio]

So far I haven’t found a way to do this yet. If you find a way do post it here so others have a way of using this as well!

[Kirthi]

@jlipps - Can you provide your thoughts on this ?

[Aleksei]

@Kirthi step by step

1. learn how appium opens driver against iOS and Android
2. with your ui create minimum number variables needed from user fill to open driver
3. make it work now all together with your program
4. once above 3 done now you can take screenshot using Appium command and page source. Using them you should be able show user device screen and identify elements user clicks with your screen.
5. now you can write your script identifying one by one user clicks. As we see user should click on screen you provide but not device itself.

[Kirthi]

@Aleksei: Thanks for the response.

Can u please explain in detail on how to mirror the screen and identify user clicked elements ??

[Aleksei]

@Kirthi you have screenshot that you show user. You have page source with all elements.

• User clicks on your screenshot.
• You getting it x,y value.
• you parse page source and identify element laying under tapped x,y
• you writing some AI logic how to tap this element or stupid xPath direct link (which changes from device to device or even android version)

[Kirthi]

@Aleksei: Thanks for the detailed description.

I have two questions -

1. How to find co-ordinates when user clicks on the mobile device ?
2. Is there any other option to find element using co-ordinates other than parsing entire Page source ?

I found an option to find elements using co-ordinates using Java Script but that works only for Web Applications, need a solution for Native apps

[Aleksei]

@Kirthi

1. user should click on screenshot you take from device and show to user. Not from device.
2. no idea. Never saw. Guess the onle way. You can look at Appium desktop code also.

[Kirthi]

Many thanks @Aleksei.