Would you like to run the Firefox OS Gaia-UI Endurance Tests, but don’t know how? This blog post will have you up and running in no time!
WARNING: Running the Gaia-UI Endurance Tests will result in data being ERASED from the Firefox OS device, including data on the device’s microSD card. Therefore, if there is any data on your Firefox OS device and microSD card that you wish to keep, be sure to back it up first before running the Gaia-UI Endurance Tests. Consider yourself warned!
Thanks to the help from Dave Hunt, the Gaia-UI Endurance Tests have now been merged into the main Gaia Github Repository (yay!). If you’re new to the Gaia-UI Endurance Tests, as a prerequisite you may wish to read my previous Gaia-UI Endurance Tests post (however please disregard the previous code location mentioned therein).
- Ubuntu 12.04 (or better) x64
- A Firefox OS device ALREADY FLASHED with an ENGINEERING build of Firefox OS B2G-18 V1-Train (1.1)
- ADB (Android SDK Platform Tool) is installed, and the environment variable ADB_PATH points to your ADB location
- Your Ubuntu instance is configured to support the USB connection to the Firefox OS device; to verify, with the device connected, open an Ubuntu terminal and type ‘adb logcat’ and see if it connects
Step 1: Clone the Gaia Repository
The Gaia-UI Endurance Tests are located in the v1-train branch of the Mozilla Github Gaia repository. Assuming that you haven’t done so already, the first step is to clone that branch of the repo. In a new Ubuntu terminal:
git clone -b v1-train https://github.com/mozilla-b2g/gaia.git
You may want to go grab a coffee and come back in five minutes.
Step 2: Run the GaiaTest Setup
The Gaia-UI Endurance tests are built upon the GaiaTest framework (which uses Marionette). The next step is to run the setup script to install GaiaTest and all of the required dependencies. You may wish to create a new virutal environment to use with the Gaia-UI Endurance Tests. If you don’t, you may need to use ‘sudo’ while running the setup command. In your terminal:
python setup.py develop
Step 3: Set Test Vars and Acknowledge Risks
GaiaTest uses a special file where you can set certain variables that are required for test configuration. For example, to tell the device which WiFi network it should use. Before running the Gaia-UI Endurance Tests, you must setup the test vars file. In your Ubuntu terminal:
cp testvars_template.json mytestvars.json
In the mytestvars.json file:
- If the Firefox OS device has a SIM card, add the corresponding phone number in the phone number entry in the “carrier” section, ie. “phone_number”: “5551234567”,
- Also add the same phone number for the “remote_phone_number” entry
- In the “wifi” section, add the SSID for the Wifi network that the Firefox OS device is to connect to during the tests, ie. “ssid”: “Wifi Network Name”
As mentioned in the big red warning above, running the Gaia-UI Endurance tests will result in data being erased from the Firefox OS device and microSD card. This is to ensure that the tests start cleanly each time. For example, running a test on a device that already has 10,000 contacts will have very different memory value results vs running the same test on a device with no existing contacts. In order to run the tests, you must acknowledge that you are aware of this data loss risk. To acknowledge the risks:
- Add this entry to the same test vars file (ie. as the first entry in the list): “acknowledged_risks”: true,
If the risks are not acknowledged in the test vars file, the tests will not run.
Step 4: Connect the Device to WiFi
Next, power on your Firefox OS device. After it boots up, unlock the screen and go into the Wifi settings, then manually select and connect to the Wifi network that you want the device to use for the tests. Ensure the Wifi settings indicate the device is connected to Wifi before continuing.
Step 5: Connect to USB and ADB Forward the Device
Attach the Firefox OS device to USB. If you’re using an Ubuntu VM, after attaching the device, ensure the VM sees the device and connects to it; in the VM select the “VM=>Removable Devices=>Your Device=>Connect” menu item and wait for the device to connect to the VM.
Then tell ADB to forward the device port to GaiaTest, by issuing the following in your Ubuntu terminal:
adb forward tcp:2828 tcp:2828
Note: If you are using the Firefox OS Leo device, you must first tell ADB to be the root user, by issuing the following:
adb forward tcp:2828 tcp:2828
Step 6: Run a Test
Now you’re ready to actually try running a test. To run the “add_contacts” endurance test, with a single iteration, use the following command line at the terminal:
gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=1 --checkpoint=1 --restart tests/endurance/test_endurance_add_contact.py
If you get a “connection refused” error it means the device USB connection didn’t work. Just repeat connecting to the device, in Step 6 above and try again.
The Firefox OS device b2g process will restart, then the “add_contact” endurance test will run with a single iteration. If you watch the Firefox OS device, you’ll see the device UI being manipulated by Marionette. After the test finishes, a memory checkpoint will be performed. Note, another update since my last posting: The Gaia-UI Endurance tests now grab the Firefox OS device’s b2g process RSS value for the memory use checkpoint (instead of the V-SIZE value).
The test result will be displayed in the terminal window. Note that this result doesn’t include the b2g process memory value; this value is stored in a text file that was created at the time of the checkpoint. To see the resulting b2g process memory value, in your same terminal:
This “suite_summary” file will contain the average b2g process memory use (RSS) value, averaged from all of the test checkpoints (in our example there was only one checkpoint anyway).
There are two other files in the “checkpoints/” folder. The “checkpoint_add_contact_(datetime)_summary.log file contains a summary for the test run. This includes the number of iterations, all of the RSS value readings from each checkpoint, the average RSS value, etc. The third file in “checkpoints” is the “checkpoint_add_contact_(datetime).log” file. This file contains the raw data received from each of the device checkpoints, from which the summary files were produced.
Command Line Options
Here is a description of the GaiaTest command line options that you can use when running the Gaia-UI Endurance Test(s):
--type The type of test being run, always use “--type=b2g” for this
--address The address where the device port is forwarded, when you issued the ADB forward command; just use “--address=localhost:2828″
--testvars The path and name of the file containing the test vars, as described in Step 3 above
--iterations The number of times to repeat each test case/test; i.e. 100 iterations of the “add_contact” test will result in 100 new address book entries
--checkpoints After every N iterations a b2g process memory checkpoint will be grabbed from the device; by default a checkpoint will be performed after all iterations are complete
--restart The Firefox OS device’s b2g process will be restarted before each new test (not each iteration) begins (i.e. a soft reset)
As an example, to run the “add_contact” test and have it perform 100 test case iterations, grabbing a device memory checkpoint after every 10 iterations, and restarting the b2g process before the test begins, you would use the following command line:
gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=100 --checkpoint=10 --restart tests/endurance/test_endurance_add_contact.py
If you wish to run the entire Gaia-UI Endurance test suite, instead of specifying a single test on the command line, for the test name just point to the manifest file, like this:
gaiatest --type=b2g --address=localhost:2828 --testvars=mytestvars.json --iterations=1 --checkpoint=1 --restart tests/endurance/manifest.ini
However, note that running the entire test suite, with 100 iterations of each test (and a checkpoint after every 10 iterations) takes approximately 16 hours.
Now you should be all set and have the Gaia-UI Endurance Tests up and running on your local machine/device. If you have any questions, ideas for test additions, or are interested in contributing to this effort, feel free to contact me!