Tuesday, February 3, 2009

gLitePROOF has been upgraded

Version 2.0.6 was the last release of gLitePROOF.
The gLitePROOF project has been upgraded. It is a "PROOF on Demand" (PoD) project now.

“PROOF on Demand” is a set of utilities, that allows to start a PROOF cluster at user request, on a batch farm or on the Grid. It provides a plug-in based system, which allows to use different job submission frontends, such as LSF or gLite WMS. Main components of “PROOF on demand” are the PROOFAgent and the PAConsole. PROOFAgent provides the communication layer between the xrootd redirector/PROOF master on the client machine and the PROOF workers on the batch or Grid machines, possibly behind a firewall. PAConsole provides a user-friendly GUI, and also allows to easily manage PROOF worker job submissions to different systems, which can later function as one uniform cluster. Installation is simple and doesn't require administrator rights, and all the processes run only in user space. “PROOF on Demand” gives users, who don't have a static PROOF cluster at their institute, the possibility to enjoy the full power of interactive analysis with PROOF.

Tuesday, September 2, 2008

gLitePROOF 2.0.6

The gLitePROOF v.2.0.6 is out.
This release should make gLitePROOF a lot more user-friendly, it contains many new features and bug fixes. This release was tested against major HEPCG gLite Grid sites, such as FZK, DESY, UNI-Hannover, Wupertal and more.

I am very sorry for the delay of this version, but additional testtings were required.

Download
User's manual
Project's Trac (Wiki, Tickets, Source Browser)

Release Notes:

gLitePROOF
Release 2.0.6 2008-09-02

- this version is based on ROOT 5.18 (see: workers are dropped if ROOT 5.20 or ROOT 5.21.02 are used - [Trac Ticket #72] )
- implemented a part of [Trac Ticket #53].
Since r1963 there is a possibility to use xrootd pools on WNs.
When users can't use external xrootd clusters or any other data source,
they may want to use local WNs as pools. In this case gLitePROOF makes UI
as a redirector and all connected workers as servers. And one can use xrdcp
to distribute data files on the cluster. Example:
xrdcp dstarmb0.root root://depc218.gsi.de:20094//tmp/
Where depc218.gsi.de - is a gLitePROOF UI and 20094 is a port for xrootd (1094 - is a default one).
In the current implementation there is only one major restriction, that a port,
which the user has assigned for xrootd should be opened on WNs.
In the next version of PROOFAgent there will a proxy provided,
so that this restriction will be voided.
- documentation has been updated.
- minor bugs have been fixed.
- Added an ability to download a PROOF log files from worker nodes. [Trac Ticket #77]
- using GAW v.3.2.0.1974, PROOFAgent v. 1.0.5.1977 and PAConsole 1.0.5 (see apps' ReleaseNotes files for more details).

glite-api-wrapper
Release 3.2.0 2008-08-20

- The project has been moved to Eclipse 3.4 "Ganymede"
- Fixed typos.
- Fixed the build of unit tests. There was a hard coded path to XERCES-C.
- Added JobLogInfo methods to the CJobManager class. Actually there are two JobLogInfo methods.
* The first one returns a container of LB Events of a given job.
* The second one formats LB Events and streams this information to a given ostream object.
- MiscCommon: added a new function-helper. It helps to retrieve user's home directory by the name of a user.
- MiscCommon: the smart_path function has been improved to handle such a case like:
~user_name/../../ (see its documentation for more information)

PROOFAgent
Release 1.0.5 2008-08-20

- Fixed typos.
- The project has been moved to Eclipse 3.4 "Ganymede".
- Fixed an improper behaviour when PROOFAgent workers fail to connect to XROOTD on Grid WNs [Trac Ticket #76].

PAConsole
Release 1.0.5 2008-08-30

- The project has been moved to Eclipse 3.4 "Ganymede".
- Fixed many typos.
- Show message box if PAConsole user submits jobs but server is not running [Trac Ticket #70].
- Added a possibility to retrieve jobs logging information from the gLite LB.
Now using a context menu of the Grid jobs list one can get a full log of the selected job. [Track Ticket #65]
- It is now possible to submit (with different jdl files) and monitor several jobs simultaneously.
Parametric jobs (gLitePROOF default jdl) monitored as well.
Every time user clicks the submit button a new job will be submitted and added to the monitoring.
In this case users can add more workers to a gLitePROOF cluster on the fly and keep tracking all of them.
Users also can remove job from monitoring using context menu of the jobs list control.
Remove is not supported for children jobs, only single jobs or parents can be removed from the monitoring. [Track Ticket #71]
- Optimizations in jobs' monitoring, which led to reductions of calls to gLite LB.
- Added a Preferences page. Users now are able to adjust PAConsole settings. Currently the following settings are available:
* Job Status update interval
* Workers update interval
more will come... [Track Ticket #73].
- PAConsole wasn't using the defined jdl file, if a user changed jdl file name manually (without Browse button).
- GUI fixes.
- Minor bugs fixing.

+ many minor and major BUGs have been fixed in all modules (see Project's Trac for more details).

The date of the next version release will be announced a bit later.

Friday, June 13, 2008

gLitePROOF 2.0.5

I am pleased to announce the release of gLitePROOF version 2.0.5.
Download
User's manual
Project's Trac (Wiki, Tickets, Source Browser)

Release Notes:

gLitePROOF
Release 2.0.5 2008-06-13
- gLitePROOF.sh - a grid job script, its error and signal handling has been improved [Trac Ticket #52].
- added the latest PAConsole.
- documentation has been updated.
- using GAW v.3.1.0.1880, PROOFAgent v. 1.0.4.1883 and PAConsole 1.0.4 (see apps' ReleaseNotes files for more details).

glite-api-wrapper
Release 3.1.0 2008-06-13
- using explicitly libclassad_ns.so(a) instead of libclassad.so(a). This is a ClassAd with namespaces
(see: https://savannah.cern.ch/bugs/?22696)
- ported to gcc 4.3 and gLiteUI 3.1.13
- minimum version of Autoconf that can successfully compile GAW's configure.ac is 2.61 now
- added the BOOST serialization library
- Handling WMP exceptions while delegating credential [Trac Ticket #68]

PROOFAgent
Release 1.0.4 2008-06-13
- documentation has been updated
- minor fixes

PAConsole
Release 1.0.4 2008-06-13
- Added a context menu item (Grid page), using which one can retrieve job
output files of the selected job [Trac Ticket #48].
- Now PAConsole supports serialization [Trac Ticket #47].
It remembers users settings and, for example, the last submitted job saved too.
Which allows user to restart PAConsole and still monitor the last job.
Configuration data saved to and loaded from $GLITE_PROOF_LOCATION/etc/proofagent.cfg.xml.
- ReleaseNotes should be a part of distr [Trac Ticket #61].

+ many minor and major BUGs have been fixed in all modules (see Project's Trac for more details).

New release of the stable gLitePROOF planned for 15.08.2008.

Thursday, June 5, 2008

PAConsole serialization

Starting from gLitePROOF 2.0.5.1858 PAConsole supports serialization.
What does it mean for the users?
Exactly the following. Now PAConsole "remembers" all basic settings and next time user starts PAConsole, it will load previously saved settings. Main advantage is that users finally are able to monitor the last submitted job even after they restart PAConsole.

New release of the stable gLitePROOF planned for 15.06.2008.

Wednesday, February 27, 2008

gLitePROOF user's manual

The gLitePROOF project has just gotten its user's manual. Please meet it at the Documentation Portal of the project.

Monday, February 4, 2008

RGLite plug-in in ROOT

Now RGLite plug-in is officially a part of ROOT. You can find the latest version of the plug-in in ROOT's trunk.

Monday, January 28, 2008

glite-api-wrapper v3

Yesterday I have released GAW v.3.

Most of the attention during this sprint was concentrated on the Catalog Manager. It has been slightly redesigned and improved. Now it is way much easier to use GAW, to support it and to extend it. Usage of the gLite LFC catalog became dramatically easier with this version of GAW.
Just a couple of examples:
#1 Changing LFC current directory to "/grid/dech"
CGLiteAPIWrapper::Instance().Init();
CCatalogManager * pCatalog( &CGLiteAPIWrapper::Instance().GetCatalogManager() );
gaw_lfc_cwd cwd;
cwd.m_dir = "/grid/dech";
pCatalog->Run( cwd );
#2 or retrieving the current LFC directory
string sRetVal;
gaw_lfc_pwd pwd;
pCatalog->Run( pwd, &sRetVal );
cout << "---> Current catalog directory: " << sRetVal << endl;
#3 or list current LFC directory to a container
LFCHelper::LFCFileInfoVector_t file_list;
gaw_lfc_ls ls;
ls.m_dir = "/grid/dech";
pCatalog->Run( ls, &file_list );
or to the stdout
gaw_lfc_sls ls;
ls.m_dir = "/grid/dgtest";
cout << "---> Listing directory: " << ls.m_dir << endl;
pCatalog->Run( ls, &cout );
more examples one can fined here

Also, this version of GAW got a redesigned XML configuration file with a schema.
Additionally the version includes a number of cosmetic changes and an updated source documentation.

supported features
  • Workload Management System operations:
    • job submission - normal, DAG and parametric jobs (gLite WMProxy API),
    • smart look-up algorithm for WMP-Endpoints,
    • job status querying (gLite LB API),
    • job output retrieving (Globus GridFTP).
  • File Catalog operations (gLite/LCG LFC API):
    • smart session manager,
    • set/query the current working catalog directory,
    • list files, directories and their stats,
    • add/remove files in a catalog namespace,
    • add/remove directories,
    • add/remove replicas from a given file.
  • An executive logging.
  • Support of an external xml configuration file with according XML schema.

Friday, January 11, 2008

ROOT HTML documentation for RGLite plug-in

I've just finished the ROOT style HTML documentation for RGLite plug-in.
Only one comment: it takes really a lot more time to write, compile and support a good documentation with the ROOT Automatic HTML Documentation style than with doxygen, for example.
Now I love doxygen much more than I used to.

Anyway, the first draft of the RGLite documentation is ready.

Tuesday, January 8, 2008

New XML schema for the GAW configuration

glite-api-wrapper (GAW) has just gotten a new XML Schema for the configuration file, starting from r1630. Documentation pages could be found here (a XML Schema documentation).

BTW, from now on we use a Xerces-C DOMBuilder interface to validate and parse GAW configurations.
DOMBuilder is an interface introduced by the W3C DOM Level 3.0 Abstract Schemas and Load and Save Specification. DOMBuilder provides the "Load" interface for parsing XML documents and building the corresponding DOM document tree from various input sources.

Monday, December 31, 2007

ClassAds and gLite UI 3.1(3.0)

There is a long-living problem in gLiteUI 3.x. I decided to post this topic here since I have received many requests of help in this issue (I received them most probably because I was the first who posted the according bug #22696 in CERN Savannah).
The problem is that one needs to compile ClassAds with namespace support enabled, because gLite UI distributive contains ClassAds which compiled without support of namespaces, though some of gLite API libraries (WMSUI for example) require ClassaAds with namespace support. And if one tries to compile its application, which uses gLite WMS API it will end up with the something like:


libglite_wms_util.so.0: undefined reference to `classad::ClassAdParser::ParseClassAd(std::basic_istream >&, bool)'
libglite_wms_jdl.so.0: undefined reference to `classad::ExprList::insert(__gnu_cxx::__normal_iterator > >, classad::ExprTree)'
libglite_wms_jdl.so.0: undefined reference to `classad::ExprList::ExprList[in-charge](std::vector > const&)' libglite_wms_jdl.so.0: undefined reference to `classad::ClassAd::SameAs(classad::ExprTree const) const' libglite_wms_jdl.so.0: undefined reference to `classad::ExprList::erase(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >)'
libglite_wms_jdl.so.0: undefined reference to `classad::ExprList::erase(__gnu_cxx::__normal_iterator > >)'
libglite_wms_jdl.so.0: undefined reference to `classad::ExprList::push_back(classad::ExprTree)'
libglite_wms_jdl.so.0: undefined reference to `classad::ClassAd::EvaluateExpr(classad::ExprTree const, classad::Value&) const'
libglite_wms_jdl.so.0: undefined reference to `classad::ClassAd::find(std::basic_string, std::allocator &

Well, I didn't so far find a better way, but just compile own version of ClassAds library.

Download classads-0.9.6 from here.
tar -xzvf classads_0.9.6.tar.gz
cd classads_0.9.6
./configure --enable-namespace
make
make install
BTW, the following patch helps to compile CLASSADS with gcc-4.x (for classads-0.9.8) or you can use classads-0.9.9, where the problem is FIXED already.

--- ../classads-0.9.8.orig/classad_stl.h 2006-06-14 21:39:30.000000000 +0200
+++ classad_stl.h 2006-11-09 10:59:25.000000000 +0100
@@ -48,7 +48,7 @@
#define classad_hash_map std::hash_map
#define classad_slist std::slist
#elif defined(__GNUC__)
- #if (__GNUC__ == 3 && __GNUC_MINOR__ > 0)
+ #if ((__GNUC__ == 3 && __GNUC_MINOR__ > 0) || (__GNUC__ > 3))
#define classad_hash_map __gnu_cxx::hash_map
#define classad_slist __gnu_cxx::slist
#else
--- ../classads-0.9.8.orig/Makefile 2006-06-14 21:39:30.000000000 +0200
+++ Makefile 2006-11-10 08:19:13.000000000 +0100
@@ -76,15 +76,17 @@
shared: shared.C
$(CPlusPlus) -fPIC -shared -o libshared.so shared.C

+test: classad_version classad_unit_tester classad_functional_tester test_xml extra_tests
+ ./classad_version
+ ./classad_unit_tester -all
+ ./test_xml
+ ./extra_tests
+
install:: $(LIB_NAME) classad_functional_tester classad_unit_tester classad_version
- mkdir -p $(LIB_INSTALL)
- mkdir -p $(BIN_INSTALL)
- mkdir -p $(INCLUDE_INSTALL)
- cp $(LIB_NAME) $(LIB_INSTALL)
- cp classad_functional_tester $(BIN_INSTALL)
- cp classad_unit_tester $(BIN_INSTALL)
- cp classad_version $(BIN_INSTALL)
- cp $(HEADER_FILES) $(INCLUDE_INSTALL)
+ mkdir -p $(LIB_INSTALL) $(BIN_INSTALL) $(INCLUDE_INSTALL)
+ install -m 644 $(LIB_NAME) $(LIB_INSTALL)
+ install -m 755 classad_functional_tester classad_unit_tester classad_version $(BIN_INSTALL)
+ install -m 644 $(HEADER_FILES) $(INCLUDE_INSTALL)

clean: tidy
rm -f classad_functional_tester classad_unit_tester classad_version