Table of Contents:
See also Synthetic Scripts FAQ (Part II).
What’s the best and easiest way to create a script?
Start with the Firefox add-on Katalon Recorder. You will probably have to modify the script it generates, but WebDriver is simple. You can also use the Katalon Recorder to test your script locally and make sure it works. Here are some other bits of advice:
Who are our hosting providers?
Why do agents sometimes appear slower than my local setup that uses WebDriver?
There are several reasons why the agents may be slower than on your laptop:
Our primary goal is to provide consistent performance in a given location. Between locations, we try to keep everything consistent except the network.
When in doubt, run several sessions from the same location to confirm that the session duration is stable, and do the same from a different location to compare the results.
What do I do about confirmation emails or OTP keys?
We don't have special support at this time, but if you can write Python code to do something using the standard libraries, you can do that in
How do I handle an alert pop-up window?
One way would be to use a try-catch clause and wait for an alert as shown below:
from selenium.common.exceptions import TimeoutException # Handle an alert pop-up try: WebDriverWait(driver, 2).until(EC.alert_is_present()) alert = driver.switch_to_alert() alert.accept() #print "alert accepted" except TimeoutException: print 'No alert pop-up'
How do I handle file uploads/downloads with WebDriver?
For file uploads, you need to either download a file or generate one in your script (we have no way of letting you upload things to the sandbox at the moment). Next, set the path of the input element to the absolute path of the file to upload.
Below is a working example that uploads a generated file with random content to https://encodable.com/uploaddemo/:
As far as downloading files, you can definitely get the browser to download something, but from there you won't able to inspect the downloaded file from your script because we have no way to extract this file from the agent to the sandbox.
How do you ensure isolation between scripts/tests?
Each script is executed in a Docker container created and destroyed for each run. The script connects to a Windows agent to drive its web browser. For isolation, we clear the browser caches, history, and DNS cache between runs.
In both cases, we don't write to the disk, so if a hacker could steal a disk from the data centers we use, they might be able to recover something, but this would be a very sophisticated and expensive attack.
We baseline visual time/job time metrics, so will we baseline synthetic times by GEO as well? The use case is that baseline response-times per-GEO could vary.
We don't baseline per job per GEO currently. We do baseline the overall page times per GEO alongside the current RUM metrics.
Can a maintenance window be scheduled to disable jobs on a regular schedule (for maintenance windows for instance)?
You can't schedule a maintenance window per se, but if you know when your maintenance windows are going to be scheduled, then you can set up your schedules so that no jobs will be executed during that window. Alternatively, if you don't know ahead of time when the maintenance windows are going to be scheduled, you can always go in manually and hit the Disable button on the jobs before the maintenance starts and then re-enable them afterward.
Do you alert immediately when a page crosses a performance threshold?
We give the options of alerting after a retry that is also over the threshold, immediate alert with no retry and alerting after n measurements are over the threshold.
How are performance alert thresholds applied to scripts with multiple pages?
Each page is analyzed individually. If any of them is over the threshold, the alert is generated.
Do you confirm site availability warnings and errors before alerting?
We do confirm a warning and error when detected. We generate events, however, at every state: issue detected, confirmed, ongoing, and cleared, and give you the option of being notified of events you care about.
How many locations do you offer?
We currently have 27 locations, and we’ll be adding more based on customer demand. Using the public cloud lets us deploy anywhere we can find a good cloud provider.
How is availability computed? What does "ppm" or "parts-per-million" mean?
Availability (parts-per-million) is computed as: (# succeeded / # attempted) * 1,000,000 and represented in per million, instead of percent, because controller metrics are always integer-valued. If we had used percent, the most precise availability we could represent would be 100%, 99%, 98%. Not, for example, 99.99%.
To convert per-million to percent, just divide by 10,000. For example, if the availability (ppm) is 999,800, then the availability (%) is 99.98.
Most screens convert to percent before showing, but "raw data" screens like the metric browser show the value per-million.
To test internal applications from external locations, can we provide a list of AWS IP/Port information so they can allow these through their firewalls?
We offer three locations with static IPs that you can use to test internal applications from external locations.
What are the browser versions for Firefox and Chrome?
Firefox runs version 41.0.1, and Chrome runs version 48 (updated on 2017-03-09). Please note that browser versions are subject to change in the future.
How often do you update the browsers that you support?
Browser versions are generally locked; however, we can upgrade within 1-2 weeks as required.
Is BSM an option for monitoring browser extensions like Flash or Silverlight?
Browser extensions like Flash and Silverlight (which are being phased out) can't be monitored by stock Selenium, which is what we support. Many in the community have extended Selenium with APIs to do this. We don't currently support the use of 3rd-party libraries in our runtime.
What scripting environments are supported?
What hardware is used to run the tests in browsers?
Browser tests are run on single-core VMs with 4 GB of RAM, although the hardware may vary slightly per location. To ensure consistent performance results, the VMs only run one job at a time.
Can Selenium scripts written in Java be converted to Python for synthetic jobs?
No. There isn't a standard accepted tool that would do that, and honestly, this is just a bad idea. These scripts do have a maintenance cost, and a pure machine conversion between two languages is just going to make this more difficult. Not to mention debugging in the first place, unless things work out of the box, which is not very likely, the machine translation is going to be really hard to debug.
What Python libraries can I use in my scripts?
You can use any library that is part of the standard distribution of Python 2.7. In addition, the following libraries are available:
Can I install my own libraries to support my scripts?
Do we impose any kind of object size limits for test scripts?
There are no size limits for test scripts. Please be aware, however, that larger files will use more time (more "expensive").
Do we have a timeframe for mobile synthetics?
We do not have a timeframe for mobile yet.