Synthetic Scripts FAQ (Part I)

[#1]

Table of Contents:

• General
• Alerts
• Availability/Location
• Browsers
• Environments
• Languages/Libraries
• Limitations/Restrictions

See also Synthetic Scripts FAQ (Part II).

General

Question

What’s the best and easiest way to create a script?

Answer

Start with the Firefox add-on Katalon Recorder. You will probably have to modify the script it generates, but WebDriver is simple. You can also use the Katalon Recorder to test your script locally and make sure it works.  Here are some other bits of advice:

• Assume you will have to clean up any auto-generated code, although the generated script will work without modifications.
• Test the script locally before uploading it to our system.

---

Question

Who are our hosting providers?

Answer

We use Amazon EC2 and SoftLayer. Other providers are being considered.

---

Question

Why do agents sometimes appear slower than my local setup that uses WebDriver?

Answer

There are several reasons why the agents may be slower than on your laptop:

• If the website you are hitting is close to you, an agent that's 10,000 miles away will be slower. That's the whole point of the global agent network!
• We clear all relevant caches on the agent machines before running the tests.
• If you enable traffic shaping, it will be slower.
• We use 1 GHz, 4GB RAM Windows VMs. These are probably less powerful than your laptop, but the effect is probably small because most sites are not CPU or memory bound.
• We wait a little longer to make sure pages have loaded fully to capture performance metrics. Generally, the effect is small, but, in extreme cases, the effect may be bigger.

Our primary goal is to provide consistent performance in a given location. Between locations, we try to keep everything consistent except the network.

When in doubt, run several sessions from the same location to confirm that the session duration is stable, and do the same from a different location to compare the results.

---

Question

What do I do about confirmation emails or OTP keys?

 

Answer

We don't have special support at this time, but if you can write Python code to do something using the standard libraries, you can do that in

your script. 

---

Question

How do I handle an alert pop-up window?

 

Answer

One way would be to use a try-catch clause and wait for an alert as shown below:

from selenium.common.exceptions import TimeoutException
# Handle an alert pop-up
try:
WebDriverWait(driver, 2).until(EC.alert_is_present())
alert = driver.switch_to_alert()
alert.accept()
#print "alert accepted"
except TimeoutException:
print 'No alert pop-up'

---

Question

How do I handle file uploads/downloads with WebDriver?

Answer

For file uploads, you need to either download a file or generate one in your script (we have no way of letting you upload things to the sandbox at the moment). Next, set the path of the input element to the absolute path of the file to upload.

Below is a working example that uploads a generated file with random content to https://encodable.com/uploaddemo/:

import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
import time
 
driver.implicitly_wait(10)
 
# Generate a file with random numbers
generated_file_name = "/tmp/generatedfile.txt"
with open(generated_file_name, "w") as generated_file:
 for _ in range(500):
 generated_file.write(str(random.randint(0,9)))
 
driver.get("https://encodable.com/uploaddemo/")
 
driver.find_element_by_css_selector("input.upform_field.file.required") \
 .send_keys(generated_file_name)
driver.find_element_by_css_selector("#formfield-email_address") \
 .send_keys("email@addre.ss");
driver.find_element_by_css_selector("#formfield-first_name") \
 .send_keys("Bob");
driver.find_element_by_css_selector("#uploadbutton") \
 .click();
 
# Wait for "Your upload is complete" text to show up
upload_complete = WebDriverWait(driver, 10).until(
 EC.presence_of_element_located((By.CSS_SELECTOR, "dt.first")))
 
# Check that things worked as expected"
assert upload_complete.text == "Your upload is complete:"
 
print "File uploaded as: %s" % \
 driver.find_element_by_css_selector(
 "#fcuploadsummary > dt:nth-of-type(3) > a").text

As far as downloading files, you can definitely get the browser to download something, but from there you won't able to inspect the downloaded file from your script because we have no way to extract this file from the agent to the sandbox.

---

Question

How do you ensure isolation between scripts/tests?

 

Answer

Each script is executed in a Docker container created and destroyed for each run. The script connects to a Windows agent to drive its web browser. For isolation, we clear the browser caches, history, and DNS cache between runs.

In both cases, we don't write to the disk, so if a hacker could steal a disk from the data centers we use, they might be able to recover something, but this would be a very sophisticated and expensive attack.

---

Question

We baseline visual time/job time metrics, so will we baseline synthetic times by GEO as well?  The use case is that baseline response-times per-GEO could vary.

 

Answer

We don't baseline per job per GEO currently. We do baseline the overall page times per GEO alongside the current RUM metrics.

---

Question

Can a maintenance window be scheduled to disable jobs on a regular schedule (for maintenance windows for instance)?

Answer

You can't schedule a maintenance window per se, but if you know when your maintenance windows are going to be scheduled, then you can set up your schedules so that no jobs will be executed during that window. Alternatively, if you don't know ahead of time when the maintenance windows are going to be scheduled, you can always go in manually and hit the Disable button on the jobs before the maintenance starts and then re-enable them afterward.

---

Alerts

Question

Do you alert immediately when a page crosses a performance threshold?

 

Answer

We give the options of alerting after a retry that is also over the threshold, immediate alert with no retry and alerting after n measurements are over the threshold.

---

Question

How are performance alert thresholds applied to scripts with multiple pages?

Answer

Each page is analyzed individually. If any of them is over the threshold, the alert is generated.

---

Question

Do you confirm site availability warnings and errors before alerting?

Answer

We do confirm a warning and error when detected. We generate events, however, at every state: issue detected, confirmed, ongoing, and cleared, and give you the option of being notified of events you care about.

---

Availability/Location

Question

How many locations do you offer?

 

Answer

We currently have 27 locations, and we’ll be adding more based on customer demand. Using the public cloud lets us deploy anywhere we can find a good cloud provider.

---

Question

How is availability computed? What does "ppm" or "parts-per-million" mean?

Answer

Availability (parts-per-million) is computed as: (# succeeded / # attempted) * 1,000,000 and represented in per million, instead of percent, because controller metrics are always integer-valued. If we had used percent, the most precise availability we could represent would be 100%, 99%, 98%. Not, for example, 99.99%.

To convert per-million to percent, just divide by 10,000. For example, if the availability (ppm) is 999,800, then the availability (%) is 99.98.

Most screens convert to percent before showing, but "raw data" screens like the metric browser show the value per-million.

---

Question

To test internal applications from external locations, can we provide a list of AWS IP/Port information so they can allow these through their firewalls?

Answer

We offer three locations with static IPs that you can use to test internal applications from external locations.

---

Browsers

Question

What are the browser versions for Firefox and Chrome?

 

Answer

Firefox runs version 41.0.1, and Chrome runs version 48 (updated on 2017-03-09). Please note that browser versions are subject to change in the future.

---

Question

How often do you update the browsers that you support?

Answer

Browser versions are generally locked; however, we can upgrade within 1-2 weeks as required.

---

Question

Is BSM an option for monitoring browser extensions like Flash or Silverlight?

 

Answer

Browser extensions like Flash and Silverlight (which are being phased out) can't be monitored by stock Selenium, which is what we support. Many in the community have extended Selenium with APIs to do this.  We don't currently support the use of 3rd-party libraries in our runtime.

---

Environments

Question

What scripting environments are supported?

 

Answer

• Python  v.2.7.x. (Python v3 is not supported yet.) Python scripts < 2.7 are likely to work, but are untested.
• Selenium: We support WebDriver, which is also sometimes known as Selenium 3. The scripts should be written using the Python WebDriver API.
• We use a custom version of the Selenium Standalone Server based on version 3.0.1. Note, this is subject to change without notice.

---

Question

What hardware is used to run the tests in browsers? 

 

Answer

Browser tests are run on single-core VMs with 4 GB of RAM, although the hardware may vary slightly per location. To ensure consistent performance results, the VMs only run one job at a time.

---

Languages/Libraries

Question

Can Selenium scripts written in Java be converted to Python for synthetic jobs?

Answer

No. There isn't a standard accepted tool that would do that, and honestly, this is just a bad idea. These scripts do have a maintenance cost, and a pure machine conversion between two languages is just going to make this more difficult. Not to mention debugging in the first place, unless things work out of the box, which is not very likely, the machine translation is going to be really hard to debug. 

---

Question

What Python libraries can I use in my scripts?

 

Answer

You can use any library that is part of the standard distribution of Python 2.7. In addition, the following libraries are available:

• selenium, the library used to drive the agent browser
• Requests 
• pysftp 

---

Question

Can I install my own libraries to support my scripts?

 

Answer

Not yet.

---

Limitations/Restrictions

Question

Do we impose any kind of object size limits for test scripts?

Answer

There are no size limits for test scripts.  Please be aware, however, that larger files will use more time (more "expensive").

---

Mobile Synthetics

Question

Do we have a timeframe for mobile synthetics?

Answer

We do not have a timeframe for mobile yet.

---