WS-JDML: A Web Service Interface for Job Submission and Monitoring Stephen M

Please download to get full document.

View again

of 24
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report



Views: 3 | Pages: 24

Extension: PDF | Download: 0

Related documents
WS-JDML: A Web Service Interface for Job Submission and Monitoring Stephen MCGough William Lee London e-Science Centre Department of Computing, Imperial College…
WS-JDML: A Web Service Interface for Job Submission and Monitoring Stephen MCGough William Lee London e-Science Centre Department of Computing, Imperial College London What Services do we need to make the Grid Work? ã One of the key services required is job submission – The ability to transparently submit a job to a resource (potentially through a DRM) where it will run ã Many DRM systems exist (Condor, Globus, SGE etc…) – Each have their own way to define a job (language) – Each have their own submission mechanism (command line, API, Service) 2 The Problem ã Submitting jobs requires – Knowledge of the job definition procedure – The ability to interface with the appropriate DRM ã The Solution – One common Job description language that can be used with all resources (eg RSL) – A generic submission system for jobs ã Using community based standards that are in common use 3 Generic Job Submission JDML Web WS-JDML Services 4 Web Service ã We are using a plain “Vanilla” Web Service – Don’t rely on any proposed WS standards – Don’t need anything more than core standards for this simple service ã Developed in Java ã Our work has been deployed into the J2EE enterprise platform – This enables ã Scalability ã Fault tolerance 5 Job Description Markup Language JDML ã Originally developed from Condor ClassAds ã Developed for the European DataGrid project ã Used within the Imperial College ICENI project ã This work is now feeding into the Global Grid Forum Job Submission Description Language standardisation work ã JDML will morph to become JSDL 6 JDML (2) ã JDML documents are written in sections – What job to run – The environment to run the job in – Where to get files from – Where to send files to at the end ã JDML is strongly typed ã Consists of name/value pairs 7 JDML (3) ã Can have DRM specific sections – It must be safe to ignore this section and the job still work correctly – Seen as a set of hints to the DRM ã File transfer is defined for multiple protocols – Grid FTP, HTTP, copy etc… – Each file may have multiple of these definitions ã DRM can select the appropriate ones to use 8 WS-JDML Architecture 9 Job Submission Port Type ã Takes a JDML document describing the job to run ã Validates the JDML so that an immediate response can be given ã Validates user credentials, passed as part of the SOAP header, using WS-Security ã Job is then placed into queue before being processed into a DRM specific version and deployed locally 10 Job Submission Port Type (2) ã Various results – Unrecognised Job Term ã The JDML contains some term that the Service doesn’t understand – Invalid Job Term ã The JDML has a term which has the wrong type or an invalid value – Successful Submission ã URI to identify the job instance is returned 11 Job Monitoring Port Type ã This port provides a means to observe the current status of a job and manipulate the output transfer mechanism ã Requires the URI representing a job provided from job submission ã Current job status is returned – pending, scheduled, running, suspended, done, exit – Not all DRMs support all states 12 Job Monitoring Port Type (2) File Transfers ã Port provides the ability to – Get portions of the files specified in the JDML transferred – Override the transfer methods given in the JDML – Indicate that files should be transferred back as attachments to the SOAP document ã Allows easy monitoring of the job progress 13 Deployment ã DRM Specific Translators have been obtained from existing code within the ICENI project – These include Shell, SGE, Globus and Condor ã Web Service architecture has been deployed in Java J2EE 1.4 platform – This provides a number of support features for the services. 14 Demo ã Hopefully ã jobservice ã Need to run over SSH 15 Further Work ã Job State Transition – The ability to represent the status of a job running within a resource ã Notification – Currently to monitor a job requires the polling of the monitoring port ã Would be better if notifications to a sink service through say WS- Notification ã Job Term Semantics – Definition of job terms using natural language – No formal model makes JDML transformation error prone – Develop an Ontology for Job submission terms 16 What do you use to build your service? ã Widely Implemented Standard Specification (1pt) – Demonstrable Multiple Implementations, e.g. SOAP, WSDL ã Implemented draft specification (2pt) – Specification in standards body and supported by most/many companies. One/few implementations exist (e.g., WS-Security, BPEL) ã Implemented draft specification (3pt) – Specification in standards body but alternatives exist. Industry is divided. One/few implementations exist. (e.g., Transactions, coordination, notification, etc.). ã Implemented proposal (4pt) – An implementation of an idea, a proposal but not submitted to standards body yet (e.g., WS-Addressing, WS-Trust, etc.) ã Non-implemented proposal (5pt) – An idea that exits as a white paper, but no code and no specification details ã Concept (6pt) – An idea that exists only as power point slides!! ã TOTAL: SOAP, WSDL, WS-Security = 3 17 Service Dependencies ã What else does your service depend on (i.e. external dependencies)? – RDBMs / J2EE EJBs – Logging (Java Logging) – Message Queue (JMS) ã What does your implementation depend on? – Java – J2EE 1.4 compliant 18 AAA & Security ã What authentication mechanism do you use? – WS-Security ã What authorisation mechanism do you use? – Flexible composition of authorisation plugins. ã What accounting mechanism do you use? – Java logging ã Does service interaction need to be encrypted? ã If these are not used now, will they be in the future? 19 Exploiting the Service Architecture ã What features from your ‘plumbing’ do you use in your service? – Event notification – Meta-data 20 Service Activity ã Multiple interaction or single user? – Multiple ã Throughput (1/per day or 100/per second?) ã Typical data volume moved in ã Typical data volume moved out 21 Service Failure ã Required Reliability – Failure semantics? ã Positive ack (might need WS-ReliableMessaging) ã Required Persistence – Job entered into the queue is always persisted ã Required Availability – One of many or unique requirement 22 Required Service Management ã Remote access to: – Usage statistics – Job Progress – Job Diagnostic and repair interfaces 23 Acknowledgements ã Director: Professor John Darlington ã Research Staff: – Anthony Mayer, Nathalie Furmento – Stephen McGough, James Stanton – Yong Xie, William Lee – Marko Krznaric, Murtaza Gulamali – Asif Saleem, Laurie Young, Gary Kong ã Contact: – – e-mail: 24
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks