New generation resource broker system of ARC
Rőczei Gábor <>
NIIF Intézet
We had developed a grid resource broker in the KnowARC project which allows us to distribute computing jobs belonging to different grid middleware solutions. The whole system is based on the Hosting Environment which has the following main features:
- It uses web-service technology: HTML, SOAP, XML, WSDL
- Several platforms are supported: Linux, Mac OS X, Solaris, and Windows soon will be too
- We can use the following languages for service development: Python, C++, Java
- It consists of loadable modules
- Several security types can be use inside the container: X509Token, UsernameToken, SAMLToken
- IPv6 also supported
The broker can send jobs to the following systems: ARC0, ARC1, gLite currently. It can be easily extend with other systems, only the necessary C++ classes need to be implemented.
There are a lot of libarcclient plugins which are relevant for the broker; these are as follows: TargetGenerator, TargetRetriever, ExecutionTarget, JobDescription, Submitter. The TargetRetriever maps the information system, the TargetGenerator maps of a comuting service, the ExecutionTarget maps a queue and the Submitter task the middleware specific submission. Every middleware solutions have these components. The JobDescription class task is the translation between job descriptions, for example if we have a Job Submission Description Language (JSDL) job and we want to send it to gLite then we need to convert it to Job Description Language (JSDL). The JobDescription class supports the following job descriptions: JSDL POSIX, XRSL, JDL, and a new JSDL type, which is not standard at the moment (the OGF PGI group working on this new description).
I will give a short overview about the ARC1 broker system, its components, and some information about the problems which we have resolved.
The following broker types implementation has been finished: FastestQueueBroker (the job will be sent to that cluster where it can start soon), RandomBroker (the cluster will choose in a random way from among the resources), FastestCPUBroker (the decision is based on the Integer Component of SPEC CPU2006 benchmark type, the job will be sent to the resource available having the best benchmark value), DataBroker (a resource will be chosen which conrtains the most input data in its cache).
The last thing to present is the possibility for the grid user to develop their own, problem-specific broker in C++ and Python.