Programatically ship endpoint web browser preformance data to an AWS Elasticsearch Service endpoint. This project is currently under Development. Please check back soon.

• Web Performance Shipper
  • Table of Contents
  • Overview
  • Project Structure
  • Run This Yourself
    • Local Development
      • Without Docker
      • With Docker
    • Production
      • Without Docker
      • With Docker
  • Dependencies
  • Meet the Team

Selenium boots, and tries to fully load different websites, collects Preformance Timings listed below and shipps them to an AWS Elasticsearch Service.

Flow of Performance Timings on Web Browsers:

navigationStart -> redirectStart -> redirectEnd -> fetchStart -> domainLookupStart -> domainLookupEnd -> connectStart -> connectEnd -> requestStart -> responseStart -> responseEnd -> domLoading -> domInteractive -> domContentLoaded -> domComplete -> loadEventStart -> loadEventEnd

See below for an explanation of the files in the tree.

├── Pipfile → Pipenv requirements.
├── Pipfile.lock → Pipenv lock file.
├── README.md
├── Dockerfile
├── requirements.txt → Python requirements.
├── run.py → Run script.
└── src
    ├── __init__.py
    ├── config.py → Developer created config file.
    ├── config_template.py → Template for config.py file.
    ├── cronTab.py →
    └── seleniumDataShipper.py → Main shipper script.
└── test
    ├── __init__.py
    └── test.py → Unittest.

This data shipper is running on Python 3.7. We strongly advise the use of either Anaconda or pipenv to manage a virtual environment in which you can install the dependencies for local development.

1. Clone this repository.
2. Run pip install -r requirements.txt to install the dependencies.
3. Generate some (AWS Elasticsearch Service) keys, and put them in a src/config.py file like so:

Please use src/config_template.py file as a template.

AWS_ES_ENDPOINT = {
    'aws_access_key_id': '',
    'aws_secret_access_key': '',
    'host': '',
    'region': '',
    'service': ''
}

4. To add the Elasticsearch index and source specific details look into the src/config.py you created, and put them in like so:

Please use src/config_template.py as a reference.

ES_INDEX = {
    'index': '',
    'doc_type': '',
    'source': ''
}

5. To add websites (i.e. https://www.google.com) look into the src/config.py you created, and put them in like so:

Please use src/config_template.py as a reference.

webPage = ["https://www.google.com"]

6. Install the ChromeDriver by using the command specific to your OS:

• On macOS: brew cask install chromedriver

  • Use Homebrew to install the chrome driver by running the command: brew cask install chromedriver
    • If you have already installed the ChromeDriver once and need to update it:
      • First try brew cask install chromedriver
      • If that errors try brew cask reinstall chromedriver
      • If that also errors run rm /usr/local/bin/chromedriver and then rerun brew cask install chromedriver
  • Confirm it installed by running: chromedriver -version

• On Windows: choco install chromedriver

  • Start by installing the Chocolatey dependency manager using the Install with CMD method.
  • Then install the ChromeDriver by running: choco install chromedriver
  • Confirm it installed by running: chromedriver -version

1. Clone this repository.
2. Generate some (AWS Elasticsearch Service) keys, and put them in a src/config.py file like so:

Please use src/config_template.py file as a template.

AWS_ES_ENDPOINT = {
    'aws_access_key_id': '',
    'aws_secret_access_key': '',
    'host': '',
    'region': '',
    'service': ''
}

4. To add the Elasticsearch index and source specific details look into the src/config.py you created, and put them in like so:

Please use src/config_template.py as a reference.

ES_INDEX = {
    'index': '',
    'doc_type': '',
    'source': ''
}

5. To add websites (i.e. https://www.google.com) look into the src/config.py you created, and put them in like so:

Please use src/config_template.py as a reference.

webPage = ["https://www.google.com"]

6. Build the docker container with docker build -t aws-es-web-performance-shipper .

7. Run the docker container with docker run aws-es-web-performance-shipper python run.py --verbose=no and your off!

This data shipper is running on Python 3.7. We strongly advise the use of Anaconda to manage a virtual environment in which you can install the dependencies.

We strongly recommend that this data shipper be run on a docker instance (due to the ease of installation) on the host machine, however if you choose not to, the steps below outline the installation procedure.

1. Clone this repository.
2. Run pip install -r requirements.txt to install the dependencies.
3. Generate some (AWS Elasticsearch Service) keys, and put them in a src/config.py file like so:

Please use src/config_template.py file as a template.

AWS_ES_ENDPOINT = {
    'aws_access_key_id': '',
    'aws_secret_access_key': '',
    'host': '',
    'region': '',
    'service': ''
}

4. To add the Elasticsearch index and source specific details look into the src/config.py you created, and put them in like so:

Please use src/config_template.py as a reference.

ES_INDEX = {
    'index': '',
    'doc_type': '',
    'source': ''
}

5. To add websites (i.e. https://www.google.com) look into the src/config.py you created, and put them in like so:

Please use src/config_template.py as a reference.

webPage = ["https://www.google.com"]

6. Install the ChromeDriver by using the command for your specific OS:

• On macOS: brew cask install chromedriver

  • Use Homebrew to install the chrome driver by running the command: brew cask install chromedriver
    • If you have already installed the ChromeDriver once and need to update it:
      • First try brew cask install chromedriver
      • If that errors try brew cask reinstall chromedriver
      • If that also errors run rm /usr/local/bin/chromedriver and then rerun brew cask install chromedriver
  • Confirm it installed by running: chromedriver -version

• On Windows: choco install chromedriver

  • Start by installing the Chocolatey dependency manager using the Install with CMD method.
  • Then install the ChromeDriver by running: choco install chromedriver
  • Confirm it installed by running: chromedriver -version

7. Run ./run.py and you're off!