ULMON Architecture
From EGEE-see WIki
The described tool consists of three basic elements: push service, pull service and web application. Also, it uses a message queuing server to share the data among the push and the pull service.
The first element is the push service. It is a Java application, ran as a cron job on every N minutes on the L&B element of the Grid. It can be also placed on a separate node, in which case remote access to the L&B database should be enabled. The push service connects to the L&B database and extracts data from it. The data extraction is done incrementally, using the timestamps of the jobs’ events and limiting the maximum records queried per run. It transfers data from the tables: users, jobs, events and short_fields. Future versions might include long_fields table too. The data is then packed and sent to a messaging server using its Java API. The pull service is a non-stop running service on a separate node. It is a subscriber module to the message queue. Its main loop waits for events from the message queue. As messages are received from the queue, they are transformed and written to a local database. The database uses unique indexing to avoid duplicates, which are necessary to maintain the complete data, since the push process is a periodical one and the L&B data has timestamps only on a detail level (the events table). Using this technique mitigates the pressure from the push server, i.e. the L&B database engine, making it more available for its primary purpose. It has also an associated cron script to monitor its running and restart it in case of failure. The third element is a web based application, which represents the users’ interface to the data. The application is hosted on a Tomcat container, accessible through HTTPS and using the underlying security infrastructure for certificates. Authorized users with valid X509 certificates (signed by any EUGridPMA member authority) can access the web application. In the first version, it supports browsing through the users own jobs, showing detailed description for each of them, filtering, statistics and conversion. The user can also download the data as MXML file for offline analysis using the ProM tool. The system uses the RabbitMQ message queuing server to enable the communication between its elements. The server is an Erlang implementation of the AMQP, an advanced message queuing protocol. The implementation in Erlang guaranties high level of availability and robustness. The tool is developed using Java, MySQL connector JDBC driver and RabbitMQ Java API. The web interface uses Tomcat container and is build using JSTL tag library.
