iRefIndex Output and Statistics

From irefindex
Revision as of 15:09, 29 September 2011 by PaulBoddie (talk | contribs) (Added a mention of the mapper_tables.sql script.)

The production of output and statistics involves two separate programs: PSI_MI_TAB_Maker and BioPSI_Suplimenter.

PSI-MI Controlled Vocabulary Mapping

NoteNote

Sabry to help document this part.

In the data maintained by iRefIndex, various controlled vocabulary terms are used which do not match genuine terms defined in the molecular interaction ontology. As a result, a process is followed involving the extraction of such unrecognised terms, the curation of a mapping to replacement terms, and the processing of the maintained data to use the replacement terms.

Currently, the curation process is performed by assembling the unrecognised terms in a spreadsheet which is then modified, adding suggested replacements alongside the existing terms.

Creating a Mapping Wiki Page

A page summarising the mapping of unrecognised terms to known terms should be prepared such as the Mapping of terms to MI term ids - iRefIndex 8.0‎ page.

The cv2wiki.py script needs to be obtained. Get the program's source code from this location:

Using CVS with the appropriate CVSROOT setting, run the following command: 

cvs co bioscape/bioscape/modules/interaction/Sabry/cv2wiki.py

The CVSROOT environment variable should be set to the following for this to work: 

export CVSROOT=:ext:<username>@hfaistos.uio.no:/mn/hfaistos/storage/cvsroot

(The <username> should be replaced with your actual username.)

Running the Program

The program must be run on comma-separated value files exported from the curation spreadsheet. First, the "interaction type" and "interaction detection method" sheets must be individually exported using the following settings:

  • The field delimiter is defined to be the comma (,) character
  • Field quoting is done using the double-quote (") character

With exported files defined, for example, as cv_int_type.csv and cv_int_det_method.csv for the "interaction type" and "interaction detection method" sheet files respectively, the following command can then be run: 

python cv2wiki.py cv_int_type.csv cv_int_det_method.csv MediaWiki

The output in the above example will be written to standard output (the terminal/console). To write to a file, add a filename as an argument to the program. For example: 

python cv2wiki.py cv_int_type.csv cv_int_det_method.csv MediaWiki CVMapping

This file could potentially be uploaded to the Wiki using a tool suitable for this purpose. For a Wiki such as MoinMoin (also supported by the program), the file could potentially be copied into place with a certain degree of care.

Building PSI_MI_TAB_Maker

The PSI_MI_TAB_Maker.jar file needs to be obtained or built.

  1. Get the program's source code from this location:

    Using CVS with the appropriate CVSROOT setting, run the following command: 

    cvs co bioscape/bioscape/modules/interaction/Sabry/PSI_MI_TAB_Maker

    The CVSROOT environment variable should be set to the following for this to work: 

    export CVSROOT=:ext:<username>@hfaistos.uio.no:/mn/hfaistos/storage/cvsroot
    (The <username> should be replaced with your actual username.)
  2. Obtain the program's dependencies. This program uses the MySQL Connector/J library which can be found at the following location:
  3. Extract the dependencies: 
    tar zxf mysql-connector-java-5.1.6.tar.gz

    This will produce a directory called mysql-connector-java-5.1.6 containing a file called mysql-connector-java-5.1.6-bin.jar which should be placed in the lib directory in the PSI_Writer directory... 

      mkdir lib
  cp mysql-connector-java-5.1.6/mysql-connector-java-5.1.6-bin.jar lib/

    You may instead choose to copy the library from the BioPSI_Suplimenter/lib directory: 

      mkdir lib
  cp ../BioPSI_Suplimenter/lib/mysql-connector-java-5.1.6-bin.jar lib/

    The filenames in the above example will need adjusting, depending on the exact version of the library downloaded.

    The SHA.jar file needs copying from its build location: 

    cp ../SHA/dist/SHA.jar lib/

    Alternatively, the external libraries can also be found in the following location: 

    /biotek/dias/donaldson3/iRefIndex/External_libraries
  4. Compile the source code. In order to build the software on a computer which does not have the NetBeans IDE installed, copy the generic build file into the PSI_MI_TAB_Maker directory: 
    cp Build_files/build.xml .

    Compile and create the .jar file as follows: 

    ant jar

Preparing the MITAB Tables

Before we can run the PSI_MI_TAB_Maker program, MITAB-related tables must be created in the database. Before this may happen, some mapping tables need to be created.

NOTE: The tables described here are prepared by the cv.no.uio.biotek.Preprocess class in a step which will be documented later.

One way of ensuring that such tables exist and are suitable is to drop any existing tables within the database being built, then to copy existing tables from a previously built database: 

use <database>;
drop table mapping_intDitection;
drop table mapping_intType;
drop table mapping_partidentification;
create table mapping_intDitection like <old_database>.mapping_intDitection;
create table mapping_intType like <old_database>.mapping_intType;
create table mapping_partidentification like <old_database>.mapping_partidentification;
insert into mapping_intDitection select * from <old_database>.mapping_intDitection;
insert into mapping_intType select * from <old_database>.mapping_intType;
insert into mapping_partidentification select * from <old_database>.mapping_partidentification;

For example: 

use irefindex;
drop table mapping_intDitection;
drop table mapping_intType;
drop table mapping_partidentification;
create table mapping_intDitection like old_db.mapping_intDitection;
create table mapping_intType like old_db.mapping_intType;
create table mapping_partidentification like old_db.mapping_partidentification;
insert into mapping_intDitection select * from old_db.mapping_intDitection;
insert into mapping_intType select * from old_db.mapping_intType;
insert into mapping_partidentification select * from old_db.mapping_partidentification;

Obtaining the SQL Scripts

Get the scripts from this location:

Using CVS with the appropriate CVSROOT setting, run the following command: 

cvs co bioscape/bioscape/modules/interaction/Sabry/SQL_commands

The CVSROOT environment variable should be set to the following for this to work: 

export CVSROOT=:ext:<username>@hfaistos.uio.no:/mn/hfaistos/storage/cvsroot

(The <username> should be replaced with your actual username.)

Running the SQL Scripts

In the SQL_commands directory, one script (preprocess_for_output.sql) provides the basis for all data output, whereas two other scripts (preprocess_for_mitab.sql and TAB_MAKE.sql) together provide a large number of SQL statements for the creation of MITAB-related tables. The first two of these scripts should be run as follows and any error conditions noted: 

mysql -h <hostname> -u <username> -p -A -D <database> < preprocess_for_output.sql
mysql -h <hostname> -u <username> -p -A -D <database> < preprocess_for_mitab.sql

The final script can then be run if no errors were experienced: 

mysql -h <hostname> -u <username> -p -A -D <database> < TAB_MAKE.sql

Running PSI_MI_TAB_Maker

Run the program as follows: 

java -jar -Xms256m -Xmx256m build/jar/PSI_MI_TAB_Maker.jar <config filename>

A sample configuration file is located in the config directory. It can be copied, modified and supplied to the program.

Running BioPSI_Suplimenter to Produce Statistics

This program was already built and run in the iRefIndex Build Process. For a completed build process it should not need to be run again, but an option does exist to explicitly produce statistics for the system, should this be required. The basic details of running the program are described in Running BioPSI_Suplimenter.

Create reports

Upon selecting this option in the running program, complete the fields as previously described. The reports will be written to the designated log file directory.

Creating a Statistics Wiki Page

The reports2wiki.py file needs to be obtained. Get the program's source code from this location:

Using CVS with the appropriate CVSROOT setting, run the following command: 

cvs co bioscape/bioscape/modules/interaction/Sabry/reports2wiki.py

The CVSROOT environment variable should be set to the following for this to work: 

export CVSROOT=:ext:<username>@hfaistos.uio.no:/mn/hfaistos/storage/cvsroot

(The <username> should be replaced with your actual username.)

Running the Program

The program can be run on the report files in the log file directory as follows: 

python reports2wiki.py /home/irefindex/output/Suplimenter03052009 MediaWiki

The "prefix" of the report files should be the common part of all such files such that... 

ls /home/irefindex/output/Suplimenter03052009*

...should list the report files (and log files) produced by iRefIndex.

The output in the above example will be written to standard output (the terminal/console). To write to a file, add a filename as an argument to the program. For example: 

python reports2wiki.py /home/irefindex/output/Suplimenter03052009 MediaWiki Statistics_iRefIndex_3.0

This file could potentially be uploaded to the Wiki using a tool suitable for this purpose. For a Wiki such as MoinMoin (also supported by the program), the file could potentially be copied into place with a certain degree of care.

Creating Other Mapping Files

In addition to the MITAB data, a mapping file should be generated as follows in the SQL_commands directory: 

mysql --local-infile -h <hostname> -u <username> -p -A -D <database> < mapper_tables.sql

This will write a file called mappings.txt to the current directory.

All iRefIndex Pages

Follow this link for a listing of all iRefIndex related pages (archived and current).

Retrieved from "http://irefindex.vib.be/wiki/index.php?title=iRefIndex_Output_and_Statistics&oldid=3739"