SROE: Standing Rules Of Engagement

Installed Erlang on OS X 10.6.3 Snow Leopard

2010-04-26T09:32:00.004+02:00

I bought 'Seven Languages in Seven Weeks' and one of those languages is Erlang. The version on Fink is v.12, but the latest is version 13, so I decided to install it myself from source. I discovered the directory path for installation is not allowed to have a space in it, so I made a path with no spaces and downloaded there. First the download and unpack; OS X doesnt have wget by default, but curl is there:

curl 'http://www.erlang.org/download/otp_src_R13B04.tar.gz' -o 'otp_src_R13B04.tar.gz'

tar -xzvf otp_src_R13B04.tar.gz

curl 'http://www.erlang.org/download/otp_doc_man_R13B04.tar.gz' -o 'otp_doc_man_R13B04.tar.gz'

Create the directory for installation:

sudo mkdir -p /sw/erlang

Now configure, make and install:

cd otp_src_R13B04

./configure --prefix=/sw/erlang --enable-threads -enable-smp-support --enable-kernel-poll

make

sudo make install

Copy the documentation into place and extract it there:

cp ../otp_doc_man_R13B04.tar.gz /sw/erlang/

cd /sw/erlang

sudo tar -xzvf otp_doc_man_R13B04.tar.gz

sudo rm otp_doc_man_R13B04.tar.gz

Edit your ~/.bashrc to include the documentation and the new erlang binary in the appropriate paths; I use the 'pico' editor, but the lines you add are:

export PATH=$PATH:/sw/erlang/bin

export MANPATH=$MANPATH:/sw/erlang/man

...and source that script to get the new paths with . ~/.bashrc.

Now type 'erl' to get in the Erlang repl, and 'ctrl-C a' to exit.

W3C schema for the RFC 1123 date format

2010-04-22T17:35:00.006+02:00

Here's a quickie I didn't find anywhere: XSD for the RFC 1123 date format, as used in http headers; note its only for GMT and years 2000 -2099 (my particular case):

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation><xs:documentation>Schema for RFC 1123 date format type</xs:documentation></xs:annotation>

<xs:simpleType name="RFC1123_date">
<xs:annotation><xs:documentation>RFC 1123 date format.</xs:documentation></xs:annotation>

<xs:restriction base="xs:string">
<xs:pattern value="(Mon|Tue|Wed|Thu|Fri|Sat|Sun), [0-3][0-9] (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) 20[0-9][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9] GMT">
</xs:restriction>
</xs:simpleType>
</xs:schema>

Python http PUT request

2010-03-17T13:41:00.004+01:00

I have a CherryPy service which accepts 'PUT' requests with a named payload parameter (it's name is , surprisingly, 'payload'). From cURL, this is easy to specify with the -F option, but I couldn't find anywhere how to do this with Python. I discovered something a bit odd, which seems to be that the payload needs to be specified twice:

import urllib, httplib

import time, calendar

import base64

if __name__=='__main__':

timenow=time.gmtime()

t=time.asctime(timenow)

iov1=str(calendar.timegm(timenow))+'000000000'

iov2=str(calendar.timegm(timenow) + 1) +'000000000'

p=''+t+''

px=urllib.quote_plus(p)

pe=urllib.urlencode({"payload":p})

urlbase='putserver.cern.ch:8081'

testPut='DEVDB10/ATLAS_SCT_COMMCOND_DEV/PUTS200/TEST/PUT/Multi/timespan/'+iov1+''+iov2+"?payload="+px

test='/cooldb'

testMerge='/cooldb/merge'

print px

print urlbase

authHeader={"Authorization": "Basic "+base64.encodestring('name:pwd'), "Content-Length":str(len(px))}

#payload=px

h=httplib.HTTPConnection(urlbase)

h.request('PUT',testPut,px,authHeader)

r=h.getresponse()

print r.status, r.reason

print r.read()

h.close()

I wanted an iomanipulator to escape XML

2010-03-03T21:46:00.006+01:00

...and so began a steep learning curve. I had some strings to insert into XML being output on the cout stream, and ideally I would simply like to do something like this:

string myXml("xml&<>");

cout<<escapexml<<myXml;

..and the output should be 'escaped'.

Of course I could have made a function and simply transformed one string into another, but I was also keen to learn about the iostream framework. Here are a few ways which iterate towards the final goal

1. 'Adapt' the string

Custom inserters can be made for a given class, so by wrapping the string in a class, I cold simply make an inserter for that class:

Here is the header:

class Wrapper{

std::string &m_data;

public:

typedef std::string::value_type value_type;

Wrapper(std::string & data):m_data(data){ }

std::string &str(){

return m_data;

}

};

std::ostream &

operator << (std::ostream & os, Wrapper & s);

...and the definition for that inserter:

std::ostream &

operator<< (std::ostream & os, Wrapper & s){

XmlEscape x(os);

for_each(s.str().begin(),s.str().end(),x);

return os;

}

where XmlEscape is a functor:

class XmlEscape{

static const char * m_chars;

static const char * m_replacements[];

static const unsigned int m_nchars=5;

std::ostream & m_os;

public:

XmlEscape(std::ostream & os);

void operator()(char c);

std::ostream & ostream();

};

with implementation:

const char * XmlEscape::m_chars ="&\'\"><";

const char * XmlEscape::m_replacements[]={"&", "'", """, ">", "<"};

XmlEscape::XmlEscape(std::ostream & os):m_os(os){

//nop

}

void

XmlEscape::operator()(char c){

unsigned int i(m_nchars);

while(i--){

if (c==m_chars[i]) {

m_os<<m_replacements[i];

return;

}

m_os<<c;

}

2. Adapt the stream

WSDL 2.0 and all that

2009-07-07T17:44:00.001+02:00

I'm trying to make a directory of my web services to make them more visible (findable) and started with this:
Keith Chapman's Blog: RESTfull Mashup with WSDL 2.0 - WSO2 Mashup Server
and the referenced WSDL file he uses. I think he in turn uses the useful IBM page on this subject, and the WSDL primer, core definition and adjunct pages. Apart from that, there's not much out there on using WSDL for RESTful services.

In theory this could all also be done in RDF, or even a mixture of the two... this might provide both the functional definition in WSDL and the ontological framework ('find me all the atlas web services which give run information') in RDF. I'll start with WSDL, progress to UDDI and then try to backtrack and insert some RDF.

Sweetxml: My first WSDL 2.0 - generating Axis2 client code for a REST API

2009-07-07T17:41:00.000+02:00

Sweetxml: My first WSDL 2.0 - generating Axis2 client code for a REST API

Safari 4: Does Acid pass the Acid test?

2009-06-09T09:25:00.006+02:00

OK, so I updated to Safari 4 and yes, sure enough it passes the Acid3 test with flying colours. Must be good, then. So I tested it against my two outstanding bugs which I stumbled across:
Corrupt screen memory: Go to that link after visting another page and you'll see the new page just billboarded on top of the one you were looking at.

Try resizing the window to see ugly stuttering visual history as the window moves...

...and then there's simple (but big number) arithmetic inside XSLT. Safari just gets it wrong. Come on guys, where are those 64 bits you promised us? Heres an example of hex conversion, where the large integer here simply gets converted to zero inside safari, but the correct answer is given in Firefox.

So Acid 3 is all very well at testing the fancy new features, but I would have thought that CSS styling of Plain Old Xml would be even more fundamental than testing the new selectors.. so Acid 3 is failing my acid test, at least.

Parametrized Xslt from javascript, using prototype

2009-06-05T13:14:00.005+02:00

Here's the final product; the original page has 'a' links in a tag with id="displayableModules". The links have the form href="BadChannelsFile.xml#12345.0" where the module id is indicated after the file. This is a kind of xpointer syntax, but I remould it so the id gives the module part and then listen for 'click' events on those ids; this causes an XSLT to be run with the id as a parameter and a new window contains the SVG which results. Along the way, I experimented with various file loading mechanisms (so I made a separate loading class) but stuck with a synchronous Ajax request, although discovered that Safari requires the method to explicitly stated as 'get' in lower case. Enjoy...

var Loader = Class.create({

initialize: function(){

this.result="";

loadXml: function(docPath){

this.ajaxLoad(docPath);

ajaxLoad: function(docPath){

//bind the passed function to the original class

//the method (get) needs to be explicitly written in lowercase for the Safari browser!

new Ajax.Request(docPath, {asynchronous: false, onSuccess: this._callback.bind(this), method: "get",

onFailure: function(r){alert("load failed");}})

_callback: function(r){

this.result=r.responseXML;

doc: function(){

if (this.result==""){

alert("no doc");

}

return this.result;

}

});

var Transformer = Class.create({

initialize: function(){

this.processor = new XSLTProcessor();

this.paramNames=[];

this.paramValues=[];

this.xslPath="";

this.xmlPath="";

this.fileLoader = new Loader();

setXslt: function(xslPath){

this.xslPath = xslPath;

this.fileLoader.loadXml(xslPath);

this.stylesheet = this.fileLoader.doc();

setXml: function(docPath){

this.xmlPath = docPath;

this.fileLoader.loadXml(docPath);

this.xmlDoc = this.fileLoader.doc();

setParam: function(paramName, paramValue){

this.paramNames.push(paramName);

this.paramValues.push(paramValue);

exec: function(thisdoc){

this.processor.importStylesheet(this.stylesheet);

for (var i=0;i<this.paramValues.length;i++){

this.processor.setParameter("",this.paramNames[i], this.paramValues[i]);

}

var fragment = this.processor.transformToFragment(this.xmlDoc, thisdoc);

return fragment;

}

});

//global functions

function changeLinks(){

//take existing links in the div 'displayableModules' and add an id, and suppress the href link

var allLinks=$("displayableModules").getElementsByTagName("a");

for (var i=0;i

var thisElement=$(allLinks[i]);

var thisElementUrl=String(thisElement.href);

var moduleId="module_".concat(thisElementUrl.substring(thisElementUrl.lastIndexOf('#') + 1));

thisElement.setAttribute("id", moduleId);

thisElement.setAttribute("name", thisElementUrl.substring(0,thisElementUrl.lastIndexOf('#')));

thisElement.setAttribute("href","");

}

listenToLinks();

}

function listenToLinks(){

//take existing links in the div 'displayableModules' and listen to them...

var allLinks=$("displayableModules").getElementsByTagName("a");

for (var i=0;i

var thisElement=$(allLinks[i]);

var thisModule=String(thisElement.id);

thisElement.observe('click',transform);

}

function transform(e) {

var moduleElement=e.element();

var thisModule=moduleElement.id;

var sourceFile=moduleElement.name;

var transformer = new Transformer();

transformer.setXslt("./xsl/graph.xsl");

transformer.setXml(sourceFile);

transformer.setParam("id",thisModule.substring(7));

w=window.open("",thisModule,"width=670,height=500");

w.document.open();

w.document.write(''<span style="color: #000000"> + thisModule + </span>'

Bad channels for module ' + thisModule.substring(7) + '

');

f=transformer.exec(w.document);

w.document.getElementById('insert').appendChild(f);

w.document.close();

}

document.observe('dom:loaded', changeLinks);

Prototype and running XSLT from JavaScript

2009-06-03T11:44:00.004+02:00

I had some problem marrying the usual method of running an XSLT with prototype's library. Here's my project: Given an xml file of 'bad channel identifiers' for a module (where channels run 0-767), display the bad channels as bars on a graph in SVG in response to a user clicking on another web page link somewhere. I chose to generate the SVG using a parametrised XSLT, but I need to pass the parameters...so I'll be running the XSLT 'manually' from JavaScript instead of using the stylesheet association.

Step 1. : Perform a transform manually using a loaded stylesheet and xml. I'll load both via an Ajax request... (this is not production code, but might help someone... and note that I already loaded the prototype library)

var Transformer = Class.create({

initialize: function(){

this.processor = new XSLTProcessor();

setXslt: function(xslPath){

//need to bind, otherwise 'this' refers to the event context, not the class

new Ajax.Request(xslPath, {asynchronous: false, onSuccess: this._setXsl.bind(this),

onFailure: function(r){alert("failure to load xsl");}});

_setXsl: function(r){

this.stylesheet=r.responseXML;

if (this.stylesheet==''){

alert("Empty stylesheet");

}

this.processor.importStylesheet(this.stylesheet);

setXml: function(docPath){

//need to bind, otherwise 'this' refers to the event context, not the class

new Ajax.Request(docPath, {asynchronous: false, onSuccess: this._setXml.bind(this),

onFailure: function(r){alert("failure to load xml");}});

_setXml: function(r){

this.xmlDoc = r.responseXML;

if (this.xmlDoc==''){

alert("Empty xml doc");

}

exec: function(){

var fragment = this.processor.transformToFragment(this.xmlDoc, document);

return fragment;

}

});

function transform() {

var transformer = new Transformer();

transformer.setXslt("xsl/test.xsl");

transformer.setXml("xml/hello.xml");

f = transformer.exec();

$('insert').appendChild(f);

}

Java 'PUT' for RESTful service

2009-05-28T09:02:00.004+02:00

I was looking around for a Java example of 'PUT' for the CherryPy/COOL server, and at least came across a 'GET' at Jose Sandovals blog... but not much on the other methods. Andrea Formica (Atlas Muons) is trying to use it to upload calibrations. Here's the solution he came up with, using Apache's library:

public String coolPutCherryPyHttpClient() throws IOException, FileNotFoundException{

String res = "none";

HttpClient client = new HttpClient();

PutMethod put = new PutMethod(this.writenodeUrl);

try {

String payloaddata = " 999999. "

+ "";

client.getParams().setAuthenticationPreemptive(true);

Credentials defaultcreds = new UsernamePasswordCredentials(this.name, this.psswd);

client.getState().setCredentials(new AuthScope(serverUrl, portNumber, AuthScope.ANY_REALM), defaultcreds);

NameValuePair par = new NameValuePair();

par.setName("payload");

par.setValue(payloaddata);

NameValuePair[] params = new NameValuePair[1];

params[0] = par;

put.setQueryString(params);

// Execute the method.

int statusCode = client.executeMethod(put);

if (statusCode != HttpStatus.SC_OK) {

System.err.println("Method failed: " + put.getStatusLine());

}

// Read the response body.

byte[] responseBody = put.getResponseBody();

// Deal with the response.

// Use caution: ensure correct character encoding and is not binary data

res = new String(responseBody);

System.out.println(new String(responseBody));

} catch (HttpException e) {

System.err.println("Fatal protocol violation: " + e.getMessage());

e.printStackTrace();

} catch (IOException e) {

System.err.println("Fatal transport error: " + e.getMessage());

e.printStackTrace();

} finally {

// Release the connection.

put.releaseConnection();

}

return res;

}

Saxon XSLT 2.0 format-number problem (update!)

2009-05-27T16:33:00.007+02:00

Later: following the post below, I managed to ask on the saxon-help list about this. The reply from Michael Kay (He Should Know!)

is that this is the expected behaviour. Looking in the W3C recommendation, it refers to the JDK 1.1 DecimalFormat class, which indeed states:

Symbol Meaning

0      a digit

#      a digit, zero shows as absent

.      placeholder for decimal separator

,      placeholder for grouping separator.

;      separates formats.

-      default negative prefix.

%      multiply by 100 and show as percentage

? multiply by 1000 and show as per mille

¤ currency sign; replaced by currency symbol; if

         doubled, replaced by international currency symbol.

         If present in a pattern, the monetary decimal separator

         is used instead of the decimal separator.

X      any other characters can be used in the prefix or suffix

'      used to quote special characters in a prefix or suffix.

...so , perverse as it seems, the behaviour is as advertised.

Original post:

One of the (many) advantages of using XSLT 2.0 is its supposed ability to understand scientific notation, e.g. 1E-2 is correctly interpreted as 0.01. However the only implementation I know of for XSLT 2.0 is Saxon, so the XSLT 2.0 standard basically defines this one application. Here's a nasty I discovered while trying to format exponential numbers using Saxon 9.1.0.6: Formatting zero with

format-number(xs:double($myNumber),'#.#######')

causes an empty string to be output.

Heres a sample xml:

and an xslt:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
exclude-result-prefixes="xs"
version="2.0">
<xsl:output method="text"/>
<xsl:template match="numbers">
<xsl:apply-templates/>
</xsl:template>
<xsl:template match="test">
<xsl:value-of select="concat('Original: ',text())"/>
Format with #.#####
'<xsl:value-of select="format-number(xs:double(.), '#.#####')"/>'
Format with #.#########
'<xsl:value-of select="format-number(xs:double(.), '#.#########')"/>'
</xsl:template>
</xsl:stylesheet>

But the output is not what one would expect:

Original: 1.3
Format with #.#####
'1.3'
Format with #.#########
'1.3'

Original: 8E-4
Format with #.#####
'.0008'
Format with #.#########
'.0008'

Original: 8E-8
Format with #.#####
''
Format with #.#########
'.00000008'

Original: 0.0
Format with #.#####
''
Format with #.#########
''

Original: 0
Format with #.#####
''
Format with #.#########
''

CherryPy Caching: reprise

2009-05-26T18:21:00.003+02:00

Finally I had to turn off caching, even with my fix in place. Users reported a malformed xml being returned for certain queries, and it turned out that anything longer than the maxobjectsize was being cached and returned truncated. I'm sure there's an easy way around this, but not sure I want to spend more time on it just now.

CherryPy Caching

2009-05-15T10:01:00.008+02:00

I just tried turning on CherryPy web caching, this is one of the few things which receives only very cursory treatment in Sylvain Hellegouarch's excellent book, 'CherryPy Essentials'. I managed by incorporating the

conf = {'/':{'tools.caching.on':True}}

line in my code, however I was dismayed to discover that once on, I couldn't form a request would not get the cached result. Neither 'Cache-Control:max-age=0' nor 'Pragma: no-cache' would do it. Looking into the code (I'm using v. 3.1.1), I found the caching.py tool and indeed saw it pays pretty much no attention to the http request headers except (bizarrely, I think) the 'Vary' header. I've inserted a few lines into caching.py to fix this and submitted a trac bug, so we'll see whether this changes. The code lines are reproduced here:

#some people still use Pragma : no-cache to demand a fresh resource

pragma_no_cache=False

pragma_header=request.headers.get('Pragma')

if pragma_header:

pragma_values=[header.strip() for header in pragma_header.split(',')]

for this_value in pragma_values:

if 'no-cache' in this_value:

pragma_no_cache=True

break

#added by shaun, look at cache-control

max_acceptable_age=MemoryCache.delay

age_seconds=int(response.time - create_time)

cache_control_header=request.headers.get('Cache-Control')

if cache_control_header:

#split string on commas to get the multiple words

cache_control_values=[header.strip() for header in cache_control_header.split(',')]

#look for max-age

for this_value in cache_control_values:

if ('max-age' in this_value) and ('=' in this_value):

age_pair=[age.strip() for age in this_value.split('=')]

if age_pair[1].isdigit():

max_acceptable_age=int(age_pair[1])

break

#return if the cache is older than the acceptable age

if (age_seconds > max_acceptable_age) or pragma_no_cache:

request.cached = False

request.cacheable = True

return False

# Add the required Age header

response.headers["Age"] = str(age_seconds)

Reviews and DB

2008-06-04T09:32:00.003+02:00

I've been reviewing the InDetGlobalMonitoring code for Heidi, and trying to get more info into my 'run summary' page for the FDR. This second has not been easy, I need to understand the Tier0 database a lot better!

On the bright side, I had my SEDUX paper accepted for the SVG conference in Nuremberg in August.

Serving jnlp from PHP for WebStart applications

2008-05-22T14:01:00.004+02:00

So... I had to incorporate the TriggerTool information in the runlist summary, but it is served as a jnlp with arguments in the jnlp format. My first attempts at generating the jnlp dynamically would work either in Safari or in Firefox, but not both until I renamed the web application to 'test.php.jnlp'.. this worked but is ugly. Finally I hit on the solution: not only making the http header into 'Content-Type: application/x-java-jnlp-file' but also adding '; name=smtp.jnlp' onto the end of that. Now the file comes back as a .jnlp which is recognised by both browsers.

This last week, meanwhile has been a hectic rush as I was initially blamed for submitting broken tags into Athena. It turned out it was a subtle bug in the pixel software which went something like this:

Gaudi has a templated stream insertion operator for all types.

A PixelDcsData class also had a stream insertion operator.

A PixelDcsData class also had a single argument constructor, that argument being 'Identifier'.

Now, when trying to output 'Identifier' with stream insertion, the compiler complained of an ambiguously overloaded '<<' because it couldn't decide whether to convert Identifier to a PixelDcsData class and then use the custom '<<' or use the intrinsic templated '<<'. Constructors with a single argument act as conversion operators. To avoid this, qualify the constructor with 'explicit'.

Tier0 and other animals

2008-05-07T17:50:00.002+02:00

Managed to get the data filenames from Tier0; wrote a PHP service which returns plain text, html or xml according to the 'Accept' header sent in the request. Wrote documentation and an example application, then incorporated this into my 'run details' service.

Created an /SCT/Derived/Monitoring folder in Cool for Kazu.

Over the last week: created a Plone site at home and made this visible via dyndns as a demo of what it can do, following from a presentation on collaborative tools last week. It seems that CERN is opting for Sharepoint as a CMS; a very bad decision, given that all our data and programs are running under linux...

Absence

2008-04-23T08:34:00.002+02:00

Long time since I updated this... I missed some Wednesdays due to meetings, then was in Ringberg ... now ill !

I managed to get the SCT Configurations to read dead strips and inserted this in SiClusterizationTool for release 14.1.0 before I went to Ringberg; however that release still has not been built due to various problems. Carl Gwilliam has been testing it and patching it since then.

I volunteered to help the Muons insert their cabling into Cool, although I have to wonder about the methodology they are using: of 1.3 million table entries, 70% of them are empty.

Today I'll fix the detector mask interpretation in runlist and see where I get to ...

M6 and all that

2008-03-06T14:30:00.003+01:00

Busy week. Spent all weekend making a run portal for run information, incorporating an rss 2.0 feed. This meant lots of changes: date input for the run list, 'breadcrumb trail' in all the xsl transforms to produce the web pages, cron job producing rss feed and extra xsl to transform an rss feed to html for ajax injection into the original page. Along the way, found a nasty bug in Safari 3.0.4 concerning css styling of xml. Monday, Tuesday: introduced tagging in the CherryPy 'PUT' interface, and started debugging some problems which Else had. Finally finished this on Wednesday, and started to tidy up SCT_Clusterization, adding an IdentifierHash method in the SCT_ConditionsSummary interface and testing it. Today spent time debugging the new date input to runlist, since it turns out that runs can overlap in time intervals if they run in different partitions.

Pleas and deadlines

2008-02-27T17:12:00.002+01:00

Spent the last four days trying to write new SCT conditions services, with a new interface for all indet services. This was finally tagged yesterday, in theory for release 14 deadline at 18:00 yesterday. However it looks like there were lots of (other) problems with the nightly, so I guess the deadline shifted a bit. It all works and I wrote some example code for both the summary interface and one of the derived tools. Today I made some cabling representations for the SCT with an xsl layer displaying the cabling files, e.g. the barrel, and similar files exist for endcap C and A. Looks like Bruce is close to uploading to the official database, so maybe I have to get busy with cabling soon. Cabling had new methods injected by Nick Barlow to use the identifier hash instead of identifier, the test code needs updating.

Next: runlist needs a date input, and the trigger people want a db 'front door' page from me (when did I volunteer for that?)

SCT conditions et al.

2008-02-20T19:31:00.002+01:00

Monday I was debugging Else's code, which uses CherryPy 'PUT' from PHP. Discovered some useful things about PHP CUrl (quirks, shall we say?) such as specifying the PUT method means a file must be provided... and then my CherryPy hangs with a 'Continue 100' message...unless one uses the 'custom' http method and declares it to be 'PUT', so then its possible to use the POSTFIELDs.

Yetserday decided on new SCT and InDet conditions reporting service, so I've started to program that. Today I updated a lot of documentation (how to write services, CherryPy, and InDetConditionsDB) and continued in the implementation.

FDR and all that

2008-02-08T10:29:00.000+01:00

In the midst of the Full Dress Rehearsal, and the runlist is being used a lot; I had to make a special version just for FDR data as well. Over the last couple of weeks I've created a new cabling file and made utilities to check consistency with Dave Robinsons output, investigated (and implemented) some ways of making the bytestream converter quicker and implemented a monitor program for the CherryPy server, so it mails me if the server stops working or the machine become unreachable. This Wednesday was mostly spent investigating Kazu's problem with the DCS data access; it seems that the folders were being transferred into the database with the wrong folder dscription. I alerted Jim Cook, who fixed it pretty quickly.

Cabling, decisions, CherryPy

2008-01-23T09:34:00.000+01:00

Cabling: The cabling was slightly more of a headache than I at first thought...an error in my xsl produced empty slots in the cabling file which then made Athena crash, because I had enabled conversion exceptions in the input stream and then never trapped the exception. Both the file and the reading package are now fixed, although its clear I need better checking of the file so I made a quick C++ command line utility to do that. There is still at least one funny on disk 9c (a duplicated identifier), which probably means there's another funny I haven't spotted yet. We really need this info in the database, I've given up asking and I'm trying to find another way to get the configuration info.

Decisions: The CERN group is being reorganized and my activities straddle both camps (i.e. two different sections): the detector subsystem and the database infrastructure. So where will I end up? The detectors need software people in the systems themselves, but the software work I do also needs 'expert' direction and supervision... watch this space.

CherryPy: Documented the 'PUT' interface and made a wiki for both the GET and PUT methods. It seems the Muon people are interested in using it and Andrea Formica very quickly made a Java app which demonstrates its usage.

Happy New Year...

2008-01-16T08:55:00.000+01:00

First task was to review the Bytestream converter package and introduce some documentation. This is not complete, but certainly better. Things are starting to get more exciting now the SCT can actually be cooled and run. Yesterday I had to make a cabling file from the configuration... still no database info available; I should try to find a work around. Also implemented a multi-channel 'PUT' on the CherryPy interface for Sergey...all in all, a fairly productive start to the year.

PUT in place

2007-12-06T16:23:00.001+01:00

Reverted to allowing only 'PUT' for a pure RESTful interface, and made a demo editor (very simple) which takes the payload of a COOL folder and uses XSL to create an XHTML form for input of new parameters. The data 'PUT' method I implemented accepts either a file or a string, so it is feasible to use it with an XmlHttpRequest object to PUT a URL with the conventional ?payload=xxxxxx kind of notation. All this is protected with basic authentication over the internal network.

Else Lytken tried it out, it seems ok.

PUT, POST and CherryPy

2007-11-29T15:59:00.000+01:00

I spent yesterday implementing a 'PUT' of data to the COOL database, using an xml description for the updated data. This now works fine, and today I introduced authentication. The method I implemented first used 'PUT' , since i shall only be updating existing channels, however (unfortunately) html forms always send a file with 'POST' so I have since had to allow POST too. The only way I could 'PUT' a file to a RESTful address was with cURL. I'm now making the cherrypy server produce a form from the payload description so that the data can be edited in a dynamic way.

CSS tables, speed

2007-11-21T09:20:00.000+01:00

I need a table which has a fixed header and scrollable content. I found a couple of links: here and here, but I get strange effects in Safari: it is difficult to make the heading columns line up with the data columns...there seems to be 2px offset between Firefox and Safari, accumulating with each column. I'll finish this today so it looks ok at least in Firefox.

The RunList application is up now, link here, with run details added in the links. One worry has been speed, and this seems to be down to pyCool. A bug was filed to look into this.

Python, libxml2 and run lists

2007-11-14T19:21:00.002+01:00

So this last week I've been working on the run list as a web application. The front end is not so difficult, although I was requested to make the server side in Python; not my first choice... particularly as CERN's web server is not set up to use python as flexibly as PHP. So I run it as a python script in cgi-bin. This in turn uses my CherryPy server to retrieve xml. Then the fun begins: I am using libxml2 which is really not well documented. I'm compiling some notes on my wiki which might be helpful... difficult to believe that in the whole world there might be only a handful of programmers who have tried this.

RESTful Interest

2007-11-07T08:12:00.000+01:00

I didn't write up last Wednesday or the one before, so what has happened? For a start the server where all the CherryPy stuff was based went down (hardware fault) and I didn't have a backup of my work, so that was painful. The CherryPy server and my demo application now work again, having installed them on atlmysql02 instead. I gave a presentation yesterday on this stuff, which seems to have generated quite some interest, and I started to produce a RunList web application reproducing the functionality in Richards python script.

CherryPy done

2007-10-24T13:06:00.000+02:00

The initial stab at implementing a CherryPy server is complete, along with documentation and a sample application to show how to use it. Last Wednesday was largely spent in meetings of various kinds, for a change. Today I'm cleaning up at looking at new projects.

Slow going with CherryPy

2007-10-10T19:25:00.000+02:00

Gradually augmenting the interface put up by Georg Stach for his GWT/Java application which uses cherrypy as the RPC server interface to the Cool database. I've been slightly hampered by the fact that: The cherrypy server is only accessible on atlmysql04 and I get thrown out of this machine every time I stop the server. Eventually made an applescript to stop the server, log me back in, start the server and change to the development directory. I've been slow picking up the PyCool methods as well, and the vagaries of different code editors when editing python (aargh, spaces or tabs!! I need a python IDE!) are frustrating. Slow progress, but progress nonetheless: now I know can return proper xml entries for things like cool folder descriptions.

CherryPy et al.

2007-10-03T10:31:00.000+02:00

Past week: Made the XSLT for Luca's XML -> hierarchical XML transformation, and this seems to work on the restructured web interface (reduces the total code to about the quarter of it's original length). Got a CherryPy server going locally and served a demo python application (this is intended as a solution for a COOL database gateway). I sort of managed to get the same thing going behind an apache server, wreaking all kinds of chaos on my httpd.conf file, but the stylesheets somehow went awry.

Today I managed to get Saverio's query tacked onto Luca's original code for the interface. The next step is to make a plot appear in the same interface.

Importing XML to a web page and using ID

2007-09-26T09:49:00.000+02:00

I've been working on the web display of the online database for DCS, starting from Luca's code, and then I shall put the backend from Satoru's on it. It seems the best way (most robust) to import XML into a web page is using an Ajax method and retrieve the .xml file. However I was surprised at first to find that the 'id' attribute didn't work with 'getElementById()', until I remembered that one of the few remaining places where a DTD is preferable over a W3C schema is in defining the id attribute to be of type 'ID'. Once done, it seems to work (tiddler article in preparation).

Today I need an XSLT to go from Luca's existing Atlas.xml file format to the new menu.xml format I made, which is less wordy and separates the menu definition from the hierarchy, using xlink:href references to bind the two.

On the back burner for now: On Monday, after reviewing a few web apps in Atlas which access Cool, it became apparent that everyone is doing their own ugly thing to get the COOL environment set up before accessing the database...better would be to have a centralized service. I'll accept the received wisdom that SOAP is too wordy, and now I've downloaded and locally installed CherryPy to expose the python interface to COOL via URLs.

Metadata: Atlas has it, but it seems that we have a homegrown solution; why aren't they using RDF?

IFrame vs Div in WebDisplay

2007-09-19T09:28:00.000+02:00

I spent the last couple of days trying to work out how to use the XSLT I made for displaying a JiveXML file for a web interface. I thought I would have to load the document, get the transform from the server and then manually instantiate the XSLT processor in javascript. This turns out not to be the case; because the JiveXML is associated with a stylesheet, I need only load it into an IFrame window directly from the server. In any case, changing the content of an IFrame programmatically turns out to be difficult.

It seems that Javascript cannot, however, access a local file directly from within itself: it needs to send it to the server, which then echoes it back to the web page. I gave a talk on the web display, which seemed to go very well (if people have not seen XSLT before, its effect can seem pretty amazing). A tiddlyWiki update is in preparation...

In the mean time, Richard Hawkings has asked for some help on the databases, which is exactly what I want to do. I need to familiarise myself with PyCool and look at the AtlCool console, too (todays work).

DCS Viewer Interface

2007-09-12T09:58:00.000+02:00

I've been involved in the Database documentation review, so have a meeting this afternoon from 15:00. In the mean time, I've also been looking at Luca's code. I think that the problem is very difficult: How to generally define a navigational interface for all subsystems. We've started with an XML definition file, but I think it needs revamping. All of this has been committed in CVS now, so this should at least make versioning easier.

Testing

2007-09-05T09:14:00.000+02:00

Have to test a number of fixes:

latest cabling in 13.0.X

both InDetPrepRawDataFormation and SiClusterizationTool in 13.0.30. Unfortunately, I can only test the compilation on this last one... I still need a good routine way to test clusterization...

Compiled/tested the cabling: ok, requested tag approval

Compiled InDetPrepRawDataFormation and SiClusterizationTool (dont have a unit test: have to figure out how to introduce one). ok, requested tag approval.

Diagnosed DB problem of Auke-Pieter Colijn, updated a simple DB query program (Ajax style) for him, here.

Broadening display, bad news

2007-09-04T11:01:00.000+02:00

I managed to separate out the styles of InDetWebDisplay such that they can be edited separately and then introduced panning and zooming. Once the coordinate system had been fixed, this was fairly trivial. Unfortunately there are still people out there using internet explorer, which doesnt understand SVG (it uses a proprietary format called VML) so I may need to introduce another level of transform for them...

Today I'll fix the savannah bug about InDetCabling warnings and try a few things in VML to see what happens.

Bad News

I've been asked to take on a job which would essentially mean the end of any serious software activity. I'm not sure that my initial refusal has been accepted. :-(

Aural Display

2007-08-30T10:01:00.000+02:00

Just back from holidays, trying to get back into work. A couple of pressing things:

Update the InDetWebDisplay - separate style information into a separate stylesheet, ensure it works in Safari and Firefox, at least (see here).
Try to introduce pan and zoom to this display.
Look at Satoru's code for querying the DB, and marry this with Luca's interface.
Fix warning messages in InDetCabling.

I've also discovered how to put a file in voice XML format on lxplus such that it can be telephoned (thanks to voxeo at vxml.org). This may be just a bit of fun just now, but I can see the possibilities for DCS monitoring. I'll try an xslt which creates an aural description of an event display to start with, to see what is possible.

Part timer

2007-08-15T09:00:00.000+02:00

Had lots of time out to try to cover the kids holidays, but can report modest success in trying to read CoraCool cabling. There are still some wrinkles, but I think I'm ready to try it with the full database (the test database only has the four modules for M3/M4 running).

Today I'll look over Luca's web browser for DCS conditions and give detailed comments, and Make an XSLT for the 'NICOS' nightly xml to show to Alex Undrus as an example of what could be done.

There have been some recent problems with the clusterization package in the development nightlies due to the wrong tag being collected. Hopefully this will be fixed, I'll find out later this morning.

pm: Managed to make a stylesheet for the NICOS web page, and posted it here. Also extensively commented Luca's browser interface, which he has since updated, and provided comments on the documentation. This is quite extensive: 25 pages. The latest nightly had some problems again, finally works this evening.

e'en: Finally managed to read all the cabling data I need from the CoraCool database and insert them into the identifier map.

Good release?

2007-08-08T08:51:00.000+02:00

I've had some time off, babysitting, but also been battling the database this week, trying to read the cabling. The original database I was working with turned out to be inconsistent, have to try again today.

Yesterday was a bit frustrating because the release failed fairly fundamentally. How are we supposed to know this in a reasonably automated way so that we don't start to work with a broken release? I think I found a way, by interrogating the release web pages automatically; of course these pages are in (not very good) html, so I have to find a way to tidy them to good xml before interrogating. It turns out that 'xmllint' willl do this. I've added a tiddler on my notebook page.

Root libs and bytestreams

2007-07-25T09:59:00.000+02:00

A couple of one-off tasks today:

Make sure I can compile against root libraries in xcode in the latest version of root.
Create a bytestream readable by clusterization, for testing.

The idea with (2) would be eventually to make an xml file with a schema which describes SCT hits for a few vents (max 10) for testing various parts of the SCT code. Initially I'll make one which refers to events using rod/link notation and then finally one which can use detector coordinates, although this will need an additional cabling file.

So stages of this task (for today) are:

chop an existing bytestream file, read it
change parts of the file programatically, read it
change the file using an xml event description, read it
make an xml schema for the event description

...and for the future:

read a cabling file in the same program
change the xml schema to allow detector coordinates
read and create events using detector coordinates

Side tasks today include: looking at the OKS database to see whether we can use it, either directly or in its COOL form, and possibly looking at how to make a SOAP request to the AMI database.

Noon: Task (1) complete and written up here, under 'ROOT and XCode'.

Clusterization and dashcode

2007-07-11T09:20:00.000+02:00

Awaiting Bruce's new (COOL version 2.0) database, I'll update clusterization to rip out any reference to ChannelStatusAlg. I have already taken all the functional code from the CTB04 algorithms.

I should also make a couple of desktop widgets for detector navigation using Dashcode, since the license for this runs out on the 15th and the new replacement version looks like its part of a ~$500 budle, at least until Leopard arrives.

Over the past week I've been helping Luca to understand dynamic html pages. I'll put the tutorial material on my TiddlyWiki.

New Beginnings

2007-07-10T13:48:00.000+02:00

This blog will take over from the local wiki page at CERN, and should be used in conjunction with my TiddlyWiki notebook , the idea being that this will be a kind of diary and the notebook will keep track of the 'products' I make and the gems of information I need to keep track of.