Feature request: support Livy and SparkSubmitSsh

[snowch]

• create a vcap class to wrap vcap.json
• create new method IAE#get_credential_vcap(cluster_instance_id) : vcap if more than one credentials on cluster, return first
• create new class Livy to encapsulate livy operations
• create new class SparkSubmitSsh to encapsulate spark submit operations

class Vcap:
    def __init__(self, json_string=None, vcap_json_filename=None):
        ...

class IAE:
    def get_credential_vcap(self, cluster_instance_id):
        ...
       vcap_json = ... cf call to get credentials
       return Vcap.loads(vcap_json)

...

class SparkSubmitSsh:
    def __init__(self, vcap):
        ...

    # See yarn examples: https://github.com/snowch/demo_2710/blob/6d87fa8e94fc8256ac287ceb915a72d73e1a9211/Step%2010%20-%20Export%20Spark%20Streaming%20model%20to%20BigI.ipynb

    def submit_yarn ...  return yarn_application_id
    def yarn_status( self, yarn_application_id ) ...
    def yarn_logs( self, yarn_application_id ) ...
    def yarn_kill( self, yarn_application_id ) ...

Example 1 - if you don’t have the vcap_json

cf = CloudFoundryAPI(api_key_filename=cf_api_key_filename)
iae = IAE(cf_client=cf)
vcap = iae.get_credential_vcap( ‘cluster_12345’ )
spark_submit = spark_submit_ssh.SparkSubmitSsh(vcap)
yarn_app_id = spark_submit.submit_yarn( ... )

Example 2 - if you have the vcap_json

vcap = Vcap( ‘./vcap.json’ )
spark_submit = spark_submit_ssh.SparkSubmitSsh(vcap)
yarn_app_id = spark_submit.submit_yarn( ... )