Table of Contents
General
Alerts
Availability/Location
Browsers
Environments
Languages/Libraries
Limitations/Restrictions
Mobile Synthetics
See also Synthetic Scripts FAQ (Part II).
General
What’s the best and easiest way to create a script?
Start with the Firefox add-on Katalon Recorder. You will probably have to modify the script it generates, but WebDriver is simple. You can also use the Katalon Recorder to test your script locally and make sure it works. Here are some other bits of advice:
- Assume you will have to clean up any auto-generated code, although the generated script will work without modifications.
- Test the script locally before uploading it to our system.
Who are our hosting providers?
We use Amazon EC2 and SoftLayer. Other providers are being considered.
Why do agents sometimes appear slower than my local setup that uses WebDriver?
There are several reasons why the agents may be slower than on your laptop:
- If the website you are hitting is close to you, an agent that's 10,000 miles away will be slower. That's the whole point of the global agent network!
- We clear all relevant caches on the agent machines before running the tests.
- If you enable traffic shaping, it will be slower.
- We use 1 GHz, 4GB RAM Windows VMs. These are probably less powerful than your laptop, but the effect is probably small because most sites are not CPU or memory bound.
- We wait a little longer to make sure pages have loaded fully to capture performance metrics. Generally, the effect is small, but, in extreme cases, the effect may be bigger.
Our primary goal is to provide consistent performance in a given location. Between locations, we try to keep everything consistent except the network.
When in doubt, run several sessions from the same location to confirm that the session duration is stable, and do the same from a different location to compare the results.
What do I do about confirmation emails or OTP keys?
We don't have special support at this time, but if you can write Python code to do something using the standard libraries, you can do that in your script.
How do I handle an alert pop-up window?
One way would be to use a try-catch clause and wait for an alert as shown below:
|
from selenium.common.exceptions import TimeoutException
# Handle an alert pop-up
try:
WebDriverWait(driver, 2).until(EC.alert_is_present())
alert = driver.switch_to_alert()
alert.accept()
#print "alert accepted"
except TimeoutException:
print 'No alert pop-up'
|
How do I handle file uploads/downloads with WebDriver?
For file uploads, you need to either download a file or generate one in your script (we have no way of letting you upload things to the sandbox at the moment). Next, set the path of the input element to the absolute path of the file to upload.
Below is a working example that uploads a generated file with random content to https://encodable.com/uploaddemo/:
import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
import time
driver.implicitly_wait(10)
# Generate a file with random numbers
generated_file_name = "/tmp/generatedfile.txt"
with open(generated_file_name, "w") as generated_file:
for _ in range(500):
generated_file.write(str(random.randint(0,9)))
driver.get("https://encodable.com/uploaddemo/")
driver.find_element_by_css_selector("input.upform_field.file.required") \
.send_keys(generated_file_name)
driver.find_element_by_css_selector("#formfield-email_address") \
.send_keys("email@addre.ss");
driver.find_element_by_css_selector("#formfield-first_name") \
.send_keys("Bob");
driver.find_element_by_css_selector("#uploadbutton") \
.click();
# Wait for "Your upload is complete" text to show up
upload_complete = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "dt.first")))
# Check that things worked as expected"
assert upload_complete.text == "Your upload is complete:"
print "File uploaded as: %s" % \
driver.find_element_by_css_selector(
"#fcuploadsummary > dt:nth-of-type(3) > a").text
|
As far as downloading files, you can definitely get the browser to download something, but from there you won't able to inspect the downloaded file from your script because we have no way to extract this file from the agent to the sandbox.
How do you ensure isolation between scripts/tests?
Each script is executed in a Docker container created and destroyed for each run. The script connects to a Windows agent to drive its web browser. For isolation, we clear the browser caches, history, and DNS cache between runs.
In both cases, we don't write to the disk, so if a hacker could steal a disk from the data centers we use, they might be able to recover something, but this would be a very sophisticated and expensive attack.
We baseline visual time/job time metrics, so will we baseline synthetic times by GEO as well? The use case is that baseline response-times per-GEO could vary.
We don't baseline per job per GEO currently. We do baseline the overall page times per GEO alongside the current RUM metrics.
Can a maintenance window be scheduled to disable jobs on a regular schedule (for maintenance windows for instance)?
You can't schedule a maintenance window per se, but if you know when your maintenance windows are going to be scheduled, then you can set up your schedules so that no jobs will be executed during that window. Alternatively, if you don't know ahead of time when the maintenance windows are going to be scheduled, you can always go in manually and hit the Disable button on the jobs before the maintenance starts and then re-enable them afterward.
How can I set the screen size for my Synthetic Job?
You can set the height and width of the browser window using the set_window_size(width, height, windowHandle='current') method. For example, if you want to set the window size to 800x600 pixels, insert this method into your Synthetic Script:
driver.set_window_size(800,600)
Alerts
Do you alert immediately when a page crosses a performance threshold?
We give the options of alerting after a retry that is also over the threshold, immediate alert with no retry and alerting after n measurements are over the threshold.
How are performance alert thresholds applied to scripts with multiple pages?
Each page is analyzed individually. If any of them is over the threshold, the alert is generated.
Do you confirm site availability warnings and errors before alerting?
We do confirm a warning and error when detected. We generate events, however, at every state: issue detected, confirmed, ongoing, and cleared, and give you the option of being notified of events you care about.
Availability/Location
How many locations do you offer?
We currently have 27 locations, and we’ll be adding more based on customer demand. Using the public cloud lets us deploy anywhere we can find a good cloud provider.
How is availability computed? What does "ppm" or "parts-per-million" mean?
Availability (parts-per-million) is computed as: (# succeeded / # attempted) * 1,000,000 and represented in per million, instead of percent, because controller metrics are always integer-valued. If we had used percent, the most precise availability we could represent would be 100%, 99%, 98%. Not, for example, 99.99%.
To convert per-million to percent, just divide by 10,000. For example, if the availability (ppm) is 999,800, then the availability (%) is 99.98.
Most screens convert to percent before showing, but "raw data" screens like the metric browser show the value per-million.
To test internal applications from external locations, can we provide a list of AWS IP/Port information so they can allow these through their firewalls?
We offer three locations with static IPs that you can use to test internal applications from external locations.
Browsers
What are the browser versions for Firefox and Chrome?
Firefox runs version 41.0.1, and Chrome runs version 48 (updated on 2017-03-09). Please note that browser versions are subject to change in the future.
How often do you update the browsers that you support?
Browser versions are generally locked; however, we can upgrade within 1-2 weeks as required.
Is BSM an option for monitoring browser extensions like Flash or Silverlight?
Browser extensions like Flash and Silverlight (which are being phased out) can't be monitored by stock Selenium, which is what we support. Many in the community have extended Selenium with APIs to do this.
We don't currently support the use of 3rd-party libraries in our runtime.
Environments
What scripting environments are supported?
- Python v.2.7.x. (Python v3 is not supported yet.) Python scripts < 2.7 are likely to work, but are untested.
- Selenium: We support WebDriver, which is also sometimes known as Selenium 3. The scripts should be written using the Python WebDriver API.
- We use a custom version of the Selenium Standalone Server based on version 3.0.1. Note, this is subject to change without notice.
What hardware is used to run the tests in browsers?
Browser tests are run on single-core VMs with 4 GB of RAM, although the hardware may vary slightly per location. To ensure consistent performance results, the VMs only run one job at a time.
Languages/Libraries
Can Selenium scripts written in Java be converted to Python for synthetic jobs?
No. There isn't a standard accepted tool that would do that, and honestly, this is just a bad idea. These scripts do have a maintenance cost, and a pure machine conversion between two languages is just going to make this more difficult. Not to mention debugging in the first place, unless things work out of the box, which is not very likely, the machine translation is going to be really hard to debug.
What Python libraries can I use in my scripts?
You can use any library that is part of the standard distribution of Python 2.7. In addition, the following libraries are available:
Can I install my own libraries to support my scripts?
Not yet.
Limitations/Restrictions
Do we impose any kind of object size limits for test scripts?
There are no size limits for test scripts. Please be aware, however, that larger files will use more time (more "expensive").
Mobile Synthetics
Do we have a timeframe for mobile synthetics?
We do not have a timeframe for mobile yet.
