Cloudmesh Version 4

PREFACE

• 1. About
  • 1.1. Features
  • 1.2. Roadmap for Future Activities
  • 1.3. Contact
• 2. Contributors

INSTALLATION

• 1. Installation
  • 1.1. Prerequisites
    • 1.1.1. Installation via pip development
    • 1.1.2. Source installation for development
  • 1.2. Installation of mongod
  • 1.3. Anaconda and Conda
• 2. Quickstart
  • 2.1. Command line
  • 2.2. Interactive shell (proposed)
  • 2.3. Command scripts
  • 2.4. Cache
  • 2.5. Manual
• 3. Configuration
  • 3.1. MongoDB
  • 3.2. Compute Cloud Providers
    • 3.2.1. AWS
    • 3.2.2. Azure
    • 3.2.3. Google
    • 3.2.4. OpenStack
    • 3.2.5. Virtual Box
    • 3.2.6. SSH
    • 3.2.7. Local
    • 3.2.8. Docker
  • 3.3. Storage Providers
    • 3.3.1. AWS S3
    • 3.3.2. Azure
    • 3.3.3. Google drive
    • 3.3.4. Box
    • 3.3.5. ADD OTHERS IF MISSING
  • 3.4. Object Store
  • 3.5. Batch
  • 3.6. REST
  • 3.7. Log File (proposed)
• 4. Cloudmesh yaml file
  • 4.1. Variables
    • 4.1.1. Replacing home
• 5. Cloudmesh Database
  • 5.1. Database Decorator
  • 5.2. Database Access
  • 5.3. Creating Uniqe Names
  • 5.4. Cloudmesh Attributes
• 6. Cloudmesh Yaml file Encrytion (TODO)
  • 6.1. Generating the Key and Certificate
  • 6.2. Validate and verify the key
  • 6.3. Encryption
  • 6.4. Decryption
  • 6.5. Cloudmesh Integration
  • 6.6. Editing the Configuration file
  • 6.7. Adding information to the configuration
  • 6.8. Separating the sensitive information

COMPUTE

• 1. Virtual Machine Management
  • 1.1. Command Line and Shell Interface
  • 1.2. Uniform Parameter Management
  • 1.3. Virtual machine management
  • 1.4. Key management
  • 1.5. Security groups
  • 1.6. Command Examples
    • 1.6.1. Ping
    • 1.6.2. Check
    • 1.6.3. Refersh
    • 1.6.4. Status
    • 1.6.5. Console
    • 1.6.6. Start
    • 1.6.7. Stop
    • 1.6.8. Terminate
    • 1.6.9. Delete
  • 1.7. AWS Quickstart

STORAGE

• 1. Cloudmesh Multi Cloud Storage Interface
• 2. Cloudmesh Storage Module
  • 2.1. Requirements
  • 2.2. AWSS3 Cloudmesh Integration
    • 2.2.1. Storage functions overview
    • 2.2.2. Create dir
    • 2.2.3. Put
    • 2.2.4. Get
    • 2.2.5. Search
    • 2.2.6. List
    • 2.2.7. Delete
  • 2.3. Pytests
    • 2.3.1. Generic Tests
    • 2.3.2. Provider Specific Pytests
  • 2.4. General features
    • 2.4.1. Command Line Interface
    • 2.4.2. Programming Interface
    • 2.4.3. Pytests
  • 2.5. Virtual Directory
    • 2.5.1. Configuration
    • 2.5.2. Pytests
  • 2.6. Google drive
    • 2.6.1. Note
    • 2.6.2. Links
• 3. Object Storage
  • 3.1. Instalation for Users
  • 3.2. Instalation for Developers
  • 3.3. Cloudmesh Object Storage Interfaces
    • 3.3.1. Object Storage with ASW S3
    • 3.3.2. Objstorage Functionality
  • 3.4. Create Object Directory
  • 3.5. Put
  • 3.6. Get
  • 3.7. Search
  • 3.8. List
  • 3.9. Delete
• 4. Cloudmesh Multi Cloud Open API Interface
  • 4.1. Pytests

WORKFLOW

• 1. Infrastructure Workflow
  • 1.1. Javascript Interface (proposed)
  • 1.2. REST
  • 1.3. Resources
• 2. Jupyter Integration (proposed)
  • 2.1. API command shell access (proposed)
  • 2.2. API calls (ok)

HPC

• 1. Batch
  • 1.1. Creating a job configuration
  • 1.2. Testing the connection
  • 1.3. Running the Job
  • 1.4. Downloading the Results
  • 1.5. Cleaning the remote
  • 1.6. Get the list of the jobs and clusters
  • 1.7. Modifying the Configuration by Setting Parameters
  • 1.8. Removing jobs and clusters

cmd5

• 1. CMD5 Integartion
  • 1.1. Install

MAMUAL PAGES

MANUAL PAGES

• 1. Reference Card (proposed)
  • 1.1. Shell
  • 1.2. Shell commands that expire after a session
  • 1.3. Clouds
  • 1.4. Comet
  • 1.5. HPC
• 2. Commands
• 3. Manual Cmd5
  • 3.1. admin
  • 3.2. banner
  • 3.3. clear
  • 3.4. default
  • 3.5. echo
  • 3.6. info
  • 3.7. pause
  • 3.8. plugin
  • 3.9. q
  • 3.10. quit
  • 3.11. shell
  • 3.12. sleep
  • 3.13. stopwatch
  • 3.14. sys
  • 3.15. var
  • 3.16. version
• 4. Compute Manual Pages
  • 4.1. batch
  • 4.2. flavor
  • 4.3. image
  • 4.4. key
  • 4.5. network
  • 4.6. open
  • 4.7. secgroup
  • 4.8. ssh
  • 4.9. vbox
  • 4.10. vcluster
  • 4.11. vm
  • 4.12. workflow
• 5. Storage Manual Pages
  • 5.1. objstorage
  • 5.2. storage
  • 5.3. vdir

DATABASE OBJECTS

Database

• 1. Cloudmesh Database
  • 1.1. Virtual Machines
    • 1.1.1. Openstack
      • 1.1.1.1. Flavor
      • 1.1.1.2. Image
      • 1.1.1.3. VM
  • 1.2. Azure AzProvider
    • 1.2.1. Flavor
    • 1.2.2. Image
    • 1.2.3. VM
  • 1.3. Azure MS Azure Library Provider
    • 1.3.1. Flavor
    • 1.3.2. Image
    • 1.3.3. VM
  • 1.4. AWS Libcloud Provider
    • 1.4.1. Flavor
    • 1.4.2. Image
    • 1.4.3. VM
  • 1.5. AWS Boto3 Provider
    • 1.5.1. Flavor
    • 1.5.2. Image
    • 1.5.3. VM
  • 1.6. Storage
    • 1.6.1. AwsS3
    • 1.6.2. Box
      • 1.6.2.1. Directory
      • 1.6.2.2. File
    • 1.6.3. Azure Blob
      • 1.6.3.1. Directory
      • 1.6.3.2. File
    • 1.6.4. AWSS3 the one from cloudmesh-cloud
      • 1.6.4.1. Directory
      • 1.6.4.2. File
    • 1.6.5. AWSS3 the one from cloudmesh-objstore
      • 1.6.5.1. Directory
      • 1.6.5.2. File
    • 1.6.6. Google Drive
      • 1.6.6.1. Directory
      • 1.6.6.2. File
    • 1.6.7. Local
      • 1.6.7.1. Directory
      • 1.6.7.2. File
  • 1.7. Workflow
  • 1.8. EMR
    • 1.8.1. Cluster Listing
    • 1.8.2. Instance Listing
    • 1.8.3. Cluster Description
    • 1.8.4. Copy File Request
    • 1.8.5. File Upload
    • 1.8.6. Run File Request
    • 1.8.7. Start Cluster Request
    • 1.8.8. Stop Cluster Request
    • 1.8.9. Step List
  • 1.9. HPC
    • 1.9.1. Batch
    • 1.9.2. Queue
    • 1.9.3. Job
  • 1.10. Keys

BENCHMARKS

Database

• 1. Benchmarks
• 2. AWS EC2 VM Management
• 3. AWS S3 File Storage
• 4. Azure Blob Storage
• 5.
• 6. Benchmark results for ‘box’ Storage
• 7.
• 8. AWS EMR Benchmarking

CODE

CODE

• 1. Code Documentation
• 2. Common
  • 2.1. DEBUG
  • 2.2. Variable
  • 2.3. Util
  • 2.4. Dotdict
  • 2.5. Locations
  • 2.6. Parameter
  • 2.7. FlatDict
  • 2.8. Printer
  • 2.9. Stopwatch
  • 2.10. Console
  • 2.11. Logger
  • 2.12. Error
  • 2.13. Shell
  • 2.14. Run
  • 2.15. DB
  • 2.16. SSH
• 3. CMD5
• 4. Cloudmesh
• 5. Management
  • 5.1. Configuration
  • 5.2. Printer
  • 5.3. Names
  • 5.4. Script
  • 5.5. Debug
• 6. Mongo
  • 6.1. MongoDB
  • 6.2. Controler
• 7. Commands
• 8. Inventory
• 9. cloudmesh-storage
• 10. cloudmesh-objstorage
• 11. cloudmesh-cloud
• 12. cloudmesh-batch
• 13. cloudmesh-emr

CONTRIBUTION

• 1. Code Conventions
• 2. Code Management
• 3. Documentation Management
• 4. Version Managemt
• 5. Pytest
  • 5.1. Installation
  • 5.2. Test Specification and Execution

Cloud Provider Accounts

Cloud Accounts

• 1. Amazon Web Services (AWS) Account Creation Tutorial
  • 1.1. Step-by-Step Guide
  • 1.2. References
• 2. Azure Blob Storage and Account Creation
  • 2.1. Azure Blob Storage
  • 2.2. Azure Storage account creation
  • 2.3. References
• 3. Setting Up Your Box Account
  • 3.1. Sign up
  • 3.2. Creating an app
  • 3.3. Authentication with JWT
  • 3.4. References
• 4. Google Storage Providers
  • 4.1. Google Drive
  • 4.2. Google Docs
  • 4.3. Python Google Drive API
    • 4.3.1. Step-by-step process
  • 4.4. References
• 5. Google (What is this?)
  • 5.1. Note
  • 5.2. Links
• 6. VM Providers (outdated)
  • 6.1. General Cloud Providers Access
  • 6.2. General Interface
  • 6.3. Explicit Use with Options
  • 6.4. Vagrant
  • 6.5. AWS
    • 6.5.1. Setup and Configuration
  • 6.6. Azure
    • 6.6.1. Setup and Configuration
  • 6.7. OpenStack
    • 6.7.1. Jetstream
    • 6.7.2. Chameleon Cloud
    • 6.7.3. Cybera
    • 6.7.4. DevStack

Cloudmesh Outdated

Outdated:

• 1. Goal (outdated)
• 2. Manual: Cloudmesh Multi Service Data Access
  • 2.1. Database Providers
    • 2.1.1. Local
    • 2.1.2. MongoDB
  • 2.2. Storage Providers
    • 2.2.1. Local
    • 2.2.2. Azure Blob Storage
  • 2.3. Getting Started
• 3. Vagrant (outdated)
• 4. CM4 Details (outdated)
  • 4.1. Extra: Vargrant
  • 4.2. What we have implemented
    • 4.2.1. The Preparation for installing cloudmesh (David)
    • 4.2.2. The Configuration files and some relative function classes (Sachith)
      • 4.2.2.1. Use the Configurations file
        • 4.2.2.1.1. Getting the config object
        • 4.2.2.1.2. Getting values
    • 4.2.3. Using the Counter file
      • 4.2.3.1. Using the counter
        • 4.2.3.1.1. Incrementing and Decrementing the counter values
        • 4.2.3.1.2. Getting and Setting the counter values
    • 4.2.4. The MongoDB Database in cloudmesh (Yu)
      • 4.2.4.1. Data Scheme in MongoDB
      • 4.2.4.2. Security in MongoDB
      • 4.2.4.3. Install MongoDB Into Local
      • 4.2.4.4. Insert and Update Documents in MongoDB
    • 4.2.5. The Virtual Machine Provider
      • 4.2.5.1. Execute Command in MongoDB
    • 4.2.6. 4. The Virtual Machine Provider
    • 4.2.7. AWS VM Operations (Yu)
    • 4.2.8. Azure VM Operation (David)
    • 4.2.9. Chameleon VM Operation (Rui and Kimball)
    • 4.2.10. VM Refactor (Rui)
  • 4.3. Flask Rest API (Sachith)
    • 4.3.1. Pre-requisites
    • 4.3.2. How to run the REST API
    • 4.3.3. API
    • 4.3.4. Examples
    • 4.3.5. Dev - restricting certain ips for certain rest calls
  • 4.4. Extra: Run Command/Script in AWS
• 5. AWS cm (outdated)
  • 5.1. Code Description
    • 5.1.1. cloudmesh.yaml
    • 5.1.2. :o: Suggestion for Redesign
    • 5.1.3. awscm.py
      • 5.1.3.1. Add resources
      • 5.1.3.2. List Resources
      • 5.1.3.3. Remove Resources
      • 5.1.3.4. View Resources
      • 5.1.3.5. Copy Instances from File
      • 5.1.3.6. Copy Instances from Folder
      • 5.1.3.7. Copy Instances
      • 5.1.3.8. Delete Instances from file
      • 5.1.3.9. Delete Instances from Folder
      • 5.1.3.10. Create instances from folder
      • 5.1.3.11. Read Instances from Folder
      • 5.1.3.12. Download INstances from file
    • 5.1.4. Download instances from folder
      • 5.1.4.1. Check instances
      • 5.1.4.2. Run instances locally
      • 5.1.4.3. Run instances remotely
      • 5.1.4.4. Run local
      • 5.1.4.5. Run local
    • 5.1.5. config.py
    • 5.1.6. resource.py
    • 5.1.7. utility.py
    • 5.1.8. run.py
    • 5.1.9. advanced.py
  • 5.2. TODO - Spark
• 6. REST Service (outdated)
  • 6.1. Pre-requisites
  • 6.2. How to run the REST API
  • 6.3. API
  • 6.4. Examples
  • 6.5. Dev - restricting certain ips for certain rest calls
• 7. Virtual Cluster (in progress)
  • 7.1. Creating a Virtual Cluster and testing connections
  • 7.2. Creating a runtime-configuration
  • 7.3. Running Parallel Remote Jobs
  • 7.4. Cleaning the remote

Indices and tables

• Index
• Module Index
• Search Page