PUT in place

Reverted to allowing only 'PUT' for a pure RESTful interface, and made a demo editor (very simple) which takes the payload of a COOL folder and uses XSL to create an XHTML form for input of new parameters. The data 'PUT' method I implemented accepts either a file or a string, so it is feasible to use it with an XmlHttpRequest object to PUT a URL with the conventional ?payload=xxxxxx kind of notation. All this is protected with basic authentication over the internal network.

Else Lytken tried it out, it seems ok. 

PUT, POST and CherryPy

I spent yesterday implementing a 'PUT' of data to the COOL database, using an xml description for the updated data. This now works fine, and today I introduced authentication. The method I implemented first used 'PUT' , since i shall only be updating existing channels, however (unfortunately) html forms always send a file with 'POST' so I have since had to allow POST too. The only way I could 'PUT' a file to a RESTful address was with cURL. I'm now making the cherrypy server produce a form from the payload description so that the data can be edited in a dynamic way.

CSS tables, speed

I need a table which has a fixed header and scrollable content. I found a couple of links: here and here, but I get strange effects in Safari: it is difficult to make the heading columns line up with the data columns...there seems to be  2px offset between Firefox and Safari, accumulating with each column. I'll finish this today so it looks ok at least in Firefox.

The RunList application is up now, link here, with run details added in the links. One worry has been speed, and this seems to be down to pyCool. A bug was filed to look into this.

Python, libxml2 and run lists

So this last week I've been working on the run list as a web application. The front end is not so difficult, although I was requested to make the server side in Python; not my first choice... particularly as CERN's web server is not set up to use python as flexibly as PHP. So I run it as a python script in cgi-bin. This in turn uses my CherryPy server to retrieve xml. Then the fun begins: I am using libxml2 which is really not well documented. I'm compiling some notes on my wiki which might be helpful... difficult to believe that in the whole world there might be only a handful of programmers who have tried this.

RESTful Interest

I didn't write up last Wednesday or the one before, so what has happened? For a start the server where all the CherryPy stuff was based went down (hardware fault) and I didn't have a backup of my work, so that was painful. The CherryPy server and my demo application now work again, having installed them on atlmysql02 instead. I gave a presentation yesterday on this stuff, which seems to have generated quite some interest, and I started to produce a RunList web application reproducing the functionality in Richards python script.

CherryPy done

The initial stab at implementing a CherryPy server is complete, along with documentation and a sample application to show how to use it. Last Wednesday was largely spent in meetings of various kinds, for a change. Today I'm cleaning up at looking at new projects.

Slow going with CherryPy

Gradually augmenting the interface put up by Georg Stach for his GWT/Java application which uses cherrypy as the RPC server interface to the Cool database. I've been slightly hampered by the fact that: The cherrypy server is only accessible on atlmysql04 and I get thrown out of this machine every time I stop the server. Eventually made an applescript to stop the server, log me back in, start the server and change to the development directory. I've been slow picking up the PyCool methods as well, and the vagaries of different code editors when editing python (aargh, spaces or tabs!! I need a python IDE!) are frustrating. Slow progress, but progress nonetheless: now I know can return proper xml entries for things like cool folder descriptions. 

CherryPy et al.

Past week: Made the XSLT for Luca's XML -> hierarchical XML transformation, and this seems to work on the restructured web interface (reduces the total code to about the quarter of it's original length). Got a CherryPy server going locally and served a demo python application (this is intended as a solution for a COOL database gateway). I sort of managed to get the same thing going behind an apache server, wreaking all kinds of chaos on my httpd.conf file, but the stylesheets somehow went awry.

Today I managed to get Saverio's query tacked onto Luca's original code for the interface. The next step is to make a plot appear in the same interface.

Importing XML to a web page and using ID

I've been working on the web  display of the online database for DCS, starting from Luca's code, and then I shall put the backend from Satoru's on it. It seems the best way (most robust) to import XML into a web page is using an Ajax method and retrieve the .xml file. However I was surprised at first to find that the 'id' attribute didn't work with 'getElementById()', until I remembered that one of the few remaining places where a DTD is preferable over a W3C schema is in defining the id attribute to be of type 'ID'. Once done, it seems to work (tiddler article in preparation).

Today I need an XSLT to go from Luca's existing Atlas.xml file format to the new menu.xml format I made, which is less wordy and separates the menu definition from the hierarchy, using xlink:href references to bind the two.

On the back burner for now: On Monday, after reviewing a few web apps in Atlas which access Cool, it became apparent that everyone is doing their own ugly thing to get the COOL environment set up before accessing the database...better would be to have a centralized service. I'll accept the received wisdom that SOAP is too wordy, and now I've downloaded and locally installed CherryPy to expose the python interface to COOL via URLs.

Metadata: Atlas has it, but it seems that we have a homegrown solution; why aren't they using RDF?

IFrame vs Div in WebDisplay

I spent the last couple of days trying to work out how to use the XSLT I made for displaying a JiveXML file for a web interface. I thought I would have to load the document, get the transform from the server and then manually instantiate the XSLT processor in javascript. This turns out not to be the case; because the JiveXML is associated with a stylesheet, I need only load it into an IFrame window directly from the server. In any case, changing the content of an IFrame programmatically turns out to be difficult.

It seems that Javascript cannot, however, access a local file directly from within itself: it needs to send it to the server, which then echoes it back to the web page. I gave a talk on the web display, which seemed to go very well (if people have not seen XSLT before, its effect can seem pretty amazing). A tiddlyWiki update is in preparation...

In the mean time, Richard Hawkings has asked for some help on the databases, which is exactly what I want to do. I need to familiarise myself with PyCool and look at the AtlCool console, too (todays work). 

DCS Viewer Interface

I've been involved in the Database documentation review, so have a meeting this afternoon from 15:00. In the mean time, I've also been looking at Luca's code. I think that the problem is very difficult: How to generally define a navigational interface for all subsystems. We've started with an XML definition file, but I think it needs revamping. All of this has been committed in CVS now, so this should at least make versioning easier.

Testing

Have to test a number of fixes:

latest cabling in 13.0.X

both InDetPrepRawDataFormation and SiClusterizationTool in 13.0.30. Unfortunately, I can only test the compilation on this last one... I still need a good routine way to test clusterization...

Compiled/tested the cabling: ok, requested tag approval

Compiled InDetPrepRawDataFormation and SiClusterizationTool (dont have a unit test: have to figure out how to introduce one). ok, requested tag approval.

Diagnosed DB problem of Auke-Pieter Colijn, updated a simple DB query program (Ajax style) for him, here.

Broadening display, bad news

I managed to separate out the styles of InDetWebDisplay such that they can be edited separately and then introduced panning and zooming. Once the coordinate system had been fixed, this was fairly trivial. Unfortunately there are still people out there using internet explorer, which doesnt understand SVG (it uses a proprietary format called VML) so I may need to introduce another level of transform for them...

Today I'll fix the savannah bug about InDetCabling warnings and try a few things in VML to see what happens.

Bad News

I've been asked to take on a job which would essentially mean the end of any serious software activity. I'm not sure that my initial refusal has been accepted. :-(

Aural Display

Just back from holidays, trying to get back into work. A couple of pressing things: 

• Update the InDetWebDisplay - separate style information into a separate stylesheet, ensure it works in Safari and Firefox, at least (see here).
• Try to introduce pan and zoom to this display.
• Look at Satoru's code for querying the DB, and marry this with Luca's interface.
• Fix warning messages in InDetCabling.

I've also discovered how to put a file in voice XML format on lxplus such that it can be telephoned (thanks to voxeo at vxml.org). This may be just a bit of fun just now, but I can see the possibilities for DCS monitoring. I'll try an xslt which creates an aural description of an event display to start with, to see what is possible.

Part timer

Had lots of time out to try to cover the kids holidays, but can report modest success in trying to read CoraCool cabling. There are still some wrinkles, but I think I'm ready to try it with the full database (the test database only has the four modules for M3/M4 running).

Today I'll look over Luca's web browser for DCS conditions and give detailed comments, and Make an XSLT for the 'NICOS' nightly xml to show to Alex Undrus as an example of what could be done.

There have been some recent problems with the clusterization package in the development nightlies due to the wrong tag being collected. Hopefully this will be fixed, I'll find out later this morning.

pm: Managed to make a stylesheet for the NICOS web page, and posted it here. Also extensively commented Luca's browser interface, which he has since updated, and provided comments on the documentation. This is quite extensive: 25 pages. The latest nightly had some problems again, finally works this evening.

e'en: Finally managed to read all the cabling data I need from the CoraCool database and insert them into the identifier map.

Good release?

I've had some time off, babysitting, but also been battling the database this week, trying to read the cabling. The original database I was working with turned out to be inconsistent, have to try again today.

Yesterday  was a bit frustrating because the release failed fairly fundamentally. How are we supposed to know this in a reasonably automated way so that we don't start to work with a broken release? I think I found a way, by interrogating the release web pages automatically; of course these pages are in (not very good) html, so I have to find a way to tidy them to good xml before interrogating. It turns out that 'xmllint' willl do this. I've added a tiddler on my notebook page.

Root libs and bytestreams

A couple of one-off tasks today: 

1. Make sure I can compile against root libraries in xcode in the latest version of root.
2. Create a bytestream readable by clusterization, for testing.

The idea with (2) would be eventually to make an xml file with a schema which describes SCT hits for a few vents (max 10) for testing various parts of the SCT code. Initially I'll make one which refers to events using rod/link notation and then finally one which can use detector coordinates, although this will need an additional cabling file.

So stages of this task (for today) are:

• chop an existing bytestream file, read it
• change parts of the file programatically, read it
• change the file using an xml event description, read it
• make an xml schema for the event description

...and for the future:

• read a cabling file in the same program
• change the xml schema to allow detector coordinates
• read and create events using detector coordinates

Side tasks today include: looking at the OKS database to see whether we can use it, either directly or in its COOL form, and possibly looking at how to make a SOAP request to the AMI database.

Noon: Task (1) complete and written up here, under 'ROOT and XCode'.

Clusterization and dashcode

Awaiting Bruce's new (COOL version 2.0) database, I'll update clusterization to rip out any reference to ChannelStatusAlg. I have already taken all the functional code from the CTB04 algorithms.

I should also make a couple of desktop widgets for detector navigation using Dashcode, since the license for this runs out on the 15th and the new replacement version looks like its part of a ~$500 budle, at least until Leopard arrives.

Over the past week I've been helping Luca to understand dynamic html pages. I'll put the tutorial material on my TiddlyWiki.

New Beginnings

This blog will take over from the local wiki page at CERN, and should be used in conjunction with my TiddlyWiki notebook , the idea being that this will be a kind of diary and the notebook will keep track of the 'products' I make and the gems of information I need to keep track of.