SetupX

Datafilehandling

2009-06-25T17:39:00.001-07:00

Open the Datafile Handling from the Adminpage:

You will get an overview of the existing datafiles that are assigned to any of the samples in the system grouped by the type on a timescale

Select the tree view and you can see each Technology with each of the filetypes assigned to that technology (technology filetype).

The content is available in table form. First all the filetypes of a technology.

From there all files that are in the category of this filetype.
And from there to the "per sample" showing all datafiles for this sample.

At the bottom of this view you also get different options to download the data for this sample / class / experiment for each of the different filetypes or all filetypes.

How to add / maintain SOPs on SetupX

2009-06-25T16:51:00.000-07:00

Navigate to the Menu to view / modify the SOPs:
Make sure a category exists that you want to assign your SOP to. Select the list of SOPs:
If that is not the case create a new Category by defining it at the bottom of the Catgories list
Navigate to the "new SOP" window and select the Category that you wish the SOP to assign to.
Fill out the rest of the information that is asked for and then add the file that contains the actual SOP as an attachment.
When ready upload it.
But the time you uploaded the new / updated SOP it will not be "active" yet.
Active it by pressing activate now next to the time when the SOP is valid. The top bar will turn green and the SOP will be "active". If the SOP replaces an older SOP the older one will be outdated.
All SOPs that are declared public - AND that are in a category that is public can be seen right on the main website in the upper left corner which is then linked to a list of all public SOPs. This makes it easier to share SOPs.

Adding datafiles

2009-06-25T16:21:00.000-07:00

When adding old data to the respository or in case samples where not correctly recognized (loss of the gctof-log-file) the system allows to assign these later on.

Simple get two lists:
- a list containing all the samples IDs
- and a list containing the coressponding filenames (whithout any extension)

To get the list of sample IDs:
Simply open the minimalistic view of the experiment.
In the list of your experiments:

Use the icon left of XML:
You will get the minimalistic view from which you can easily extract the sample IDs.

(Best is to create an Excel sheet out of it)

Now navigate to the admin page and use the 3rd option of the file handling:

This will get you two big input fileds in which you can copy and paste the filenames and the samples IDs:

Insert the ids and filenames (without extension but with run number).
Then submit the form. Now the system will check on every possible combination and filetype and assign the samples to each sample.
(This process can take up to 10 minutes for very large datasets.)

After that you will get a status of what the system found and what files were assigned to each of the samples.

In case you forgot to process a number of samples or placed them in the incorrect location you can redo the entire process again. Only new data files will be assigned to the system.

revision 590.

2009-06-12T17:07:00.001-07:00

The Revision 590 contains a new Formobject type: StringInputfieldAjax

Reminder - in this case it is required to create new hibernate mapping files:

Run: ant generate compile deploy

This version supports ajax for instant queries.

MailAPI conflict with google API

2009-04-22T12:31:00.000-07:00

Not directly SetupX related but useful:
A conflict within the APIs from Google and Sun are causing the following error:

org.apache.commons.mail.EmailException: Sending the email to the following server failed : XXX.XXXX.com:25
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:873)
at org.apache.commons.mail.Email.send(Email.java:898)
Caused by: javax.mail.NoSuchProviderException: smtp
at javax.mail.Session.getService(Session.java:611)
at javax.mail.Session.getTransport(Session.java:541)

So finally after searching for quite a while I found the solution:
http://forums.sun.com/thread.jspa?messageID=9884528#9884528

How to move SetupX behind an SSL

2009-04-02T10:32:00.000-07:00

You can use apache mod_ssl and mod rewrite to secure setupx jboss

Here is the rewrite without SSL.

ServerAdmin XXXX
ServerName XXXX.bioinformatics.ucdavis.edu

ErrorLog /var/log/httpd/setupx_error_

log
CustomLog /var/log/httpd/setupx_access_log combined
LogLevel warn

RewriteEngine On
ProxyPreserveHost On
RewriteRule ^/(.*)$ \
http://localhost:8180/xx1/$1 [P]

You need to enable/install mod_ssl to enable SSL and you can add the following rewrite code to protect setupx in the SSL configuration section.

RewriteEngine On
ProxyPreserveHost On
RewriteRule ^/(.*)$ \
http://localhost:8180/xx1/$1 [P]

SSL can only protect a virtual host per IP address on port 443, however,
you can have multiple virtual hosts for port 80.

thanks to Zhi-Wei Lu.

note:j boss needs memory

2009-03-11T12:40:00.001-07:00

if your JBoss is hungry for more memory:

give JBoss more memory on startup:

modify /usr/share/jboss/bin/run.conf

#Specify options to pass to the Java VM. #

JAVA_OPTS="-server -Xms128m -Xmx800m -Xdebug"

BinBase export: Combine Classes for export

2009-03-11T12:38:00.002-07:00

Exporting samples to BinBase is in general done by just sending the entire set of classes of ONE experiment from SetupX to BinBase. In Order to export just a smaller subset of classes of a big experiment select the export shown below. Also if you have classes from different experiments that you want to compare in one exported file use this method.

How to export classes of multiple experiments:

load one of the experiment that contains one of the classes that you want to export
go to export and select the "Combine multiple classes from different experiments in one export before exporting them."
you will get a list of all "your" experiments.
check the experiments that contain any of the classes that you want to export.
After you got all the experiments marked navigate to the next page
you will get a matrix of all the experiments that you selected earlier with all their classes
now select the classes you want to have in the export
the next page shows you finally only the classes
give your dataset a self describing name and submit it

Resultfile

The result file will NOT replace the original datafile that might exist already.
You manually have to download it from the resultfile folder on the GCTOF drive.

G:\setupx\result (in windows)

BinBase export: Regroup the samples for the export

2009-03-11T12:38:00.001-07:00

How to export samples in a structure not matching the original experimental design.

Export samples

In order to export samples you can either regroup the samples by creating a new experiment and "move" your samples into the new generated classes or simply export the samples to BinBase in a self structured design. Advantage is that you can export it in multiple different designs. Disadvantage is that the structure that you set up for the export is only temporarily.

The way to do it:

select the export in the detail view of the experiment
choose the option "Regroup the experiment before sending it to binbase."
assign each sample to one of the groups shown on the right

in case you do not want to export the sample - remove it by using the small "trash" icon to the left of the groups

when done - hit export at the bottom of the page.

Resultfile

The result file will NOT replace the original datafile that might exist already.
You manually have to download it from the resultfile folder on the GCTOF drive.

G:\setupx\result (in windows)

update:

You can now choose from a set of predefined "group mechanisms". Like

all samples of each class together
every 1st run with every 1st, every 2nd with every 2nd, ...
every 1st sample of a class with every 1st, every 2nd of class with every 2nd, ...
group every 2nd, 3rd, 4th or 5th class with each other.

Labels: The LabPrinter

2009-03-11T12:36:00.000-07:00

The Software for the DYMO Labelwriter 400 provides a simple interface to print labels on tough-tags 3/8 inch.

Create an XLS sheet containing two colums:

first column the BIGGER id printed in the middle of the label
second column should contain additional information about the sample (name, 4rd party identifier)

Launch application:

open the browser and navigate to SetupX
at the bottom left of the loginpage is sais:
launch the application
it will show you:

Enter the name of the lab and your name.
Press print File & Select your XLS sheet.
The first column is taken from the XLSsheet and printed in the middle.
While the second column is printed below that ID.
Do not use formulas and the labels should be Strings and NOT numeric.
Enjoy your samples.

creating sequences for LTQ advion and xcalibur

2009-02-17T15:45:00.000-08:00

I added a feature in SetupX allowing to create sequences for LTQ setups.

The extension for SetupX creates sequences based on the selection of samples that are supposed to run on the machine including randomisation and offset on the plate.

SetupX actually creates two sequences in case you have a setup of a finnigan LTQ (xcalibur) and an Advion Nanomate. We are putting it into production right now.

JBoss5 Version conflict

2009-02-11T12:21:00.000-08:00

Zhiwei Lu (Lead System Administrator) from the Bioinformatiocs Core here at UC Davis just reported that there is a problem running SetupX with the newest version of the JBoss Application server.

The folder structure of the new 5series JBoss is entirely different which will cause problems with SetupX because some of the libraries are references from the JBoss.

All 4series Version have been tested and run fine.

UC Davis Fiehnlab is on SetupX 2.0

2009-02-04T18:21:00.000-08:00

Ok - this afternoon I made the switch to the new release and so far everything is stable. There is a high load on the database because I am converting to the new "datafile" objects allowing more flexibility regarding the handling of multiple technologies.

All functions are stable except some minor glitches:

the file scanners seem not to kick in automatically
the deployment of the SOAP WebService needs to be checked - it did not come up the first time.

The webfrontend is offline until tomorrow morning - by then the conversion scripts should be completed.

I will provide an update script in the next couple of days so that existing SetupX instances can be upgraded.

Java(TM) 2 (build 1.4.2_17-b06)

2009-02-04T17:02:00.001-08:00

Ok - source modified - it is now running as the last version too on a Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_17-b06). Ignore the next posting.

Java 1.4.2 is getting left behind

2009-02-04T13:24:00.000-08:00

From the 2.0 Version (opentechnologyplatform) SetupX will require a Java 1.5 JDK.

Progress on SetupX opentechnology

2009-02-03T15:59:00.000-08:00

The new release of SetupX supports multiple platforms. So how to display the progress of an experiment when not all samples necessarily run on one and the same platform. The image on top shows the new progress bar. It actually is pretty simple but I wanted to explain it really quick.
The experiment 332849 is not finished - actaully only 24% or 93 samples have been run. 24% of them ran on "any" of the technologies.
22% were actually run on the GC while 4% ran on the LCQ platform - shown by the darker green bars. The light green bars in the background show actually 26% which represents the entire progress of the experiment.
Dark green shows the progress per specific technology and light green the progress of the entire experiment.

Release Candidate 1: open technology

2009-01-09T14:56:00.000-08:00

The release of the updated version 2.0 of SetupX is about to be released. The subversion repository contains release candidate 1 of the new version.

Few comments on the 2.0 OpenTechnologyPlatform-Release:

SetupX supports any Technologies (GCMS, LCQ, ...) in your lab. It allows you to reference ANY datafiles related to samples.
Repository of Standard Operation Procedures
Speed improvement

Last stable version:
The last stable version of SetupX is now 1.93 (SVN 421).

If you are planning on upgrading to 2.0 OT:
The change requires changes in the database structure - a couple of new tables are created and new SQL functions are required on the DB-Server. So all production systems should stay in 1.93(421) until a complete update script for the mysql is released.

reading the Leco file

2008-12-01T18:01:00.000-08:00

[FYI: This information relates to the current version as of Dec 2008 - which will be replaced by the new technology independent platform.]

The current version of SetupX communicates with the Leco GC through a number of files.

seq-File containing the sequences that are loaded into Leco ChromaTOF.
the actual data files processed by the dataprocessing methods in ChromaTOF
the encrypted logfile written by ChromaTOf

Because ChromaTOF does not allow a direct communication these files are used to either put information into ChromaTOF or receive information back from the instrument.

First of all this combination allows SetupX to create sequences. These sequences are preorganized "jobs" for the GC. SetupX generates these sequences in an automated way after the operator simply selects the samples that should be included in this run. SetupX adds Quality controls, Method blanks, Blanks, ... - then randomizes the samples and adds Blanks as needed. Out of this the actual seq file is created containing all this information including all samples IDs and file names generated for each of the datafiles. This reduces the chance of human error through mistyping, misplacement, ...
The seq file is then loaded into ChromaTOF and the measurements are started.
While the machine is running it is writing a continuous logfile containing information of the last sample that was run - which not necessarily has to be a successful run. This file is parsed by the SetupX scanner. If a new successful run is found in the logfile SetupX will register this internal and mark the sample as "run".

Why is there a redundancy - "fining the datafiles" and "finding an entry in the logfile"?
The process of determining if a sample was run could be done in a more simple way by simply checking if a datafile which is specified in the seq file does exist or not. It certainly would work but time has shown that through system outages on the datastorage systems or simply through problems with the ChromaTOF software some files are not created - which would basically mean that a sample will never be marked as "run" even so the run was successful. Which would also result in an incomplete experiment.
Therefore the sample log file (which is not human readable) is read and the information if a sample was completed is gathered from there. This makes it much more reliable and easy for the operator to determine where the actual problem lies in case data files are missing.

Copy the logfile and read that instead of using the actual datafile: In order to avoid any interference between ChromaTOF and the SetupX-Filescanner I recommend creating a copy of the actual logfile every couple of minutes and parsing this using the scanner.

A copy task would be:
copy "C:\Program " Files\Leco\Pegasus\TupFiles\00\00\03\00000365.000" "G:\ucdavis\samplelog\sample-log-bart.bin

The cryptic leco file located at
"C:\Program " Files\Leco\Pegasus\TupFiles\00\00\03\00000365.000":

upload experiment related documents

2008-11-05T11:54:00.001-08:00

SetupX is the central lab system allowing you to keep track of your experiments, samples and machines. With the release 355 the system allows you in a very easy way to share the documents, files, images, ... that you created for a certain experiment. Simply upload the files and they are automaticlly shared with every of your colaborators.

Simply open the detail view of the experiment and select "Shared Documents" icon.

It will immediately show you all the documents that have been uploaded already and allow you to add you own documents.

Summary of changes in the source

2008-11-05T11:35:00.000-08:00

What has happened in the last couple of weeks in SetupX code. These are changes made since release 350:

add: Ability to easily up- and download documents that are related to a certain experiment.
fix: allows only MasterUsers to edit an experiment that has been run already. Until now it was possible for every user that got the "lab" status.
+++:Performance Optimization - major performance improvements by optimizing queries, adding second level caching, ...
fix: problems with adding custom fields assigned to each sample.
add: tree visualisation of SOPs, Technologies, Experiments
fix: label printer was optimized for "hitting" the labels more accurate
fix: minor changes

Speed...

2008-11-03T16:00:00.000-08:00

By the time SetupX was started there were only a couple of experiments in the repository and the number of users and hits was pretty limited. Since it has been filled with a good amount of data there was a strong need for improving the speed of the underlying persistence layer. I did a couple of test to determine what the best way is of improving the speed.

The graph shows a simple test that I ran. It was always the same number of queries with a brand new connection. The orange line shows the query with a standard connection without a secondlevel cache activated in hibernate while the second one - the secondlevelcache is activated but the first 10 loops of queries are run without declaring the queries as cacheable.

hqlSession.createCriteria(Message.class).setCacheable(cache).addOrder(Order.asc("id")).setMaxResults(limit).list();

It was certainly worth cause in some cases the time saved is up to 85%.

(The peak at datapoint 5 on the green line is caused by a scanner (that can not be deactivated) running in the background causing a big load for a very short moment. It shows up randomly in the graph everytime I ran the test.)

---
I ran the tests again.

publishing curated data

2008-10-22T16:53:00.001-07:00

In order to publish curated data on SetupX you have to import the data containing the curated datavalues from an Excel sheet. The following images show how this is done.

Select the datafile that contains the samples and the compounds in rows / columns or the other way round. They can be transposed later on.

The file is then uploaded and analyzed by the system. After the system is done you will now see the first 25 lines of your excel sheet. Now select all the columns and rows that are non relevant for you and remove them. By using the icon next to the row / the column.

After shrinking your sheet to the relevant data click the icon at the bottom and you will get the first 25 lines again.

First of all select the row that contains the names of the compounds. If the compound names are not in a row you have to transpose them by using the icon in the upper left corner.

After selecting the colum containing the SetupX ID and the row containg the compoundname, the system will show all valid values as a green, checked circle in those rows that the system recognizes as a known compound.
All others cells show the actual value in that field. In order to add a compound to the list of known compounds click the icon on top and it will be recognized from now on as a known compound.
When you are satisfied with the selection enter a name for this data set and submit it. From now on the data is in the system and can be queried.
The detail view of an experiment will now show:

Dymo Label Printer fixed

2008-10-15T18:48:00.001-07:00

The little helper from the lab got a little update. It kept causing problems that it did not hit the labels correctly anymore.
The labels should be easier to read now.
All SetupX Orbiter Installations will update itself automatically the next time you use them.

Dymo Label Printer

2008-10-14T16:52:00.000-07:00

Reports from Labs using the SetupX Orbiter for printing on the Dymo Label Printer say that they encounter problems where the printer does not print acuratly in the center of the labels anymore. I am working on a fix as soon as possible.

share your documents

2008-10-14T14:52:00.000-07:00

Something that has been around for a while but has never been completed. It was actually just a couple of simple JSPs missing. So now you are able to upload documents to SetupX. There documents can be of what ever kind. Meaning that you are able to upload your finished reports, drafts, ...
These files then will automatically be shared with everyone that has access to this experiment.

updating your SetupX

2008-10-10T15:45:00.000-07:00

SetupX is under development all the time. In order to get the newest features, security updates,... onto your local installation update the sourcecode to the newsest SVN version and then simly deploy it to the Application server. The following example shows how this is done.

SVN:
yes you are right - svn is very easy to update. You navigate in the folder that you need to update and simply run "svn up". It will update all files in this directory and in all sub directories.
To update the system you dont need to run the entire installer.

Below example of deploying the newest sources (command in red is what you have to run):

scholz@linux-0r9t:/usr/share/
setupx/lims/source/m4> cd /usr/share/setupx/lims/source/m4/
scholz@linux-0r9t:/usr/share/setupx/lims/source/m4> su
Password:
linux-0r9t:/usr/share/setupx/lims/source/m4 # svn up
A share/msi_sx_msi_map2.jsp
A share/sop_index.jsp
A share/sop_detail.jsp
A share/sop_create.jsp
A share/sop_mod.jsp
A share/sop_new.jsp
A share/sop_upload.jsp
U src/org/setupx/repository/Config.java
U src/org/setupx/repository/core/communication/technology/platform/TechnoCoreObject.java
U src/org/setupx/repository/core/communication/technology/platform/Datafile.java
A src/org/setupx/repository/core/communication/document
A src/org/setupx/repository/core/communication/document/sop
A src/org/setupx/repository/core/communication/document/sop/StandardOperationProcedure.java
A src/org/setupx/repository/core/communication/document/DocumentObjectException.java
A src/org/setupx/repository/core/communication/document/DocumentObject.java
U src/org/setupx/repository/core/util/File.java
U src/org/setupx/util/Debugger.java
Updated to revision 350.
linux-0r9t:/usr/share/setupx/lims/source/m4 # rm -r build dist
linux-0r9t:/usr/share/setupx/lims/source/m4 # ant deploy
Buildfile: build.xml
Duplicated project name in import. Project m1 defined first in /usr/share/setupx/lims/source/m4/build_hibernate.xml and again in /usr/share/setupx/lims/source/m4/build.xml

init:
prepare_directories:
[mkdir] Created dir: /usr/share/setupx/lims/source/m4/dist
[mkdir] Created dir: /usr/share/setupx/lims/source/m4/buil
[...]

Screenshots and a Slides showing SetupX functionality.

2008-10-10T15:31:00.001-07:00

Click here to download the powerpoint presentation.

How to download SetupX

2008-10-10T14:44:00.001-07:00

Setup SetupX
The entire installation of SetupX is distributed via subversion. The Subversion repository contains the most recent stable version of SetupX. In addition you can download different prepared installations as ISO images or images for a vmware.

Downloading the newest release
In order to keep your installation up to date the sources are kept in a Subversion repository – it contains the latest version – make sure that your system is kept up to date by refreshing the sources from time to time.

SVN
Check every required data out from the repository by running (it requires that the folder /usr/share/setupx does exist already)
svn checkout http://setupx.googlecode.com/svn/trunk/ /usr/share/setupx
This will get you a local copy of all files required to run your version of SetupX including all required ontologies and taxonomies.

(this is part of the documentation - you will get a copy of the entire document when you check it out)

Download instructions

SetupX blog started

2008-10-10T14:31:00.000-07:00

Due to the fact that SetupX is released and that I can see more and more interesst in the software now, I decided to get a fast way of publishing the news / changes / comments.

run your own Test SetupX

2008-09-18T14:54:00.000-07:00

I created an image on VMWare running Ubuntu which includes all relevant packages, databases, setting to run SetupX.

Load it and start - it is full functional.

ftp://fiehnlab.ucdavis.edu/setupx/download/vmware-images/

email after the Metabolomics Conference

2008-09-11T15:03:00.000-07:00

After the metabolomics meeting with a lot of interesst in SetupX the following email went out to everyone that I talked to or that just dropped a Business Card at my poster.

SetupX - the study design database. University of California,
Davis. Metabolomics - Oliver Fiehn

Good afternoon,

Thank you for your interest in SetupX - the study design database. I
met a lot of you at the Metabolomics Society Meeting in Boston last
week and I just wanted to get back to you with this email.
First of all the poster that was shown at the conference in Boston is
available for download.
Feel free to contact me in case you need information beyond that.
http://fiehnlab.ucdavis.edu/projects/binbase_setupx/Scholz_Boston_2008.pdf

SetupX is open source:
For a couple of month SetupX has been released now as an open source
project and I am happy to see that there is interest in the project.
If you are interested in setting up an instance of SetupX in your lab
environment the following link will guide you to all relevant sources
required to set up your personal instance of SetupX. Basic knowledge
in Unix administration is enough to be able to install the system. A
manual for the installation and configuration is part of the download.
http://setupx.fiehnlab.ucdavis.edu:8080/m1/readme.jsp

Coming up in the next release:
The current version of SetupX that is for download right now is
supporting Leco machines (including samples scheduling, randomization,
tracking,...). When talking to a lot of the people there was quite
some interest in setting SetupX up in an environment with machines
other than Leco GC TOFs. The next version which is being tested right
now adds functionality allowing more instruments to be connected to
the software. This will give you the ability to define the machine in
SetupX by yourself and keep track of all different data files to each
sample/experiment in one and the same interface.

Feel free to contact me for any information,

Martin Scholz

- Metabolomics -
UC Davis Genome Center
GBSF Building
451 East Health Sciences Drive
Davis (CA) 95616-8816

Poster at the Metabolomics Meeting 2008 (Boston)

2008-09-02T15:08:00.000-07:00

SetupX is open source!

2008-06-01T14:49:00.000-07:00

At the ASMS 2008 SetupX is released as an open source project.

SetupX publication

2007-01-01T14:47:00.000-08:00

Metabolomic databases are useless without accurate description of the biological study design and accompanying metadata reporting on the laboratory workflow from sample preparation to data processing. Here we report on the implementation of a database system that enables investigators to detail and set up a biological experiment, and that also steers laboratory workflows by direct access to the data acquisition instrument.[...]