Table of content:
Some glossary :
The first part describes the hierarchy created by the default installation script and the second part describes the hierarchy of the running servers.
You may skip the first part is you only want to test TOMUSS
The hierarchy created by the installation tree insure that a running server can be stopped and an older version of the server can be quickly started without data loss.
The installation script may use 2 directories, one for the server and one for the backup server. They store exactly the same things (except temporary and log files). The database stay synchronized in the two directories.
If the installation script is used, the following directories are symbolic links to the version independant directories.
The top level directories
The core of the code
When a TOMUSS process is started it loads its plugins. The received requests are dispatched to the fittest plugin. The most important attributes of a plugin are.
Do not modify plugins.py source to add plugins, use a redefine to add plugins
Do not modify global state with plugins because it may cause problems if the plugin is reloaded while the server is running.
It defines the pattern of the request. For example: {Y}/{S}/{U}/resume
Here are the components of the URLs and where they are stored in the request object in the server side.
| Component | Attribute name | Comment |
|---|---|---|
| {Y} | the_year | The year of interest |
| {S} | the_semester | The semester of interest |
| {U} | the_ue | The course of interest |
| {P} | the_page | The page of the table |
| {?} | something | Not important |
| {I} | the_student | A student identifier |
| {_I} | the_student | A student identifier prefixed by _ |
| {*} | the_path | The remaining of the path |
| {=} | Remove any /=.... from the path | |
| { } | the_time | The navigator request time (avoid caching) |
Plugin usage can be restricted to a list of user groups using config_plugins table, user groups are defined with the config_acls table. The default authorized group list contains the value of the group parameter.
The possible values of the decision attributes are None to indicate that it is not important, True to indicate that session must match the attribute, False to indicate that session must not match the attribute.
The action attributes are:
Some of the plugins are used both in TOMUSS and 'suivi' servers.
The ``link and position´´ defines the function generating the HTML content to insert in an home page. The position specify the name of the box where to insert the code.
_INCLUDE_ xxx_doc_plugins.htmlIf you want to replace a builtin TOMUSS plugin by yours, you must remove the standard one before defining your replacement. To delete the 'picture' plugin:
plugin.plugins.remove(plugin.get("picture"))
The client perform action on the server by inserting image objects in the HTML code. The returned image give the user feedback (green: ok, orange: waiting server, red: unauthorized, violet: bug...)
The client does not poll server to retrieve changes, but the server send javascript fragment when there is an update to send.
The client load the documents as usual. But if the document is a table:
The table data is stored as a Python module, the module source is only modified by doing an append to prevent any data loss. File size is checked before and after append to insure that the data was really saved.
All the TOMUSS data are stored in two files, so two identical hierarchies are managed.
The columns and lines are identified by a key usualy created as: page_id + '_' + number. With this scheme, two identical keys are not possible.
The functions to modify the table are:
_INCLUDE_ xxx_data.htmlInformations about ABJ and DA are stored in one key file per student. The parameters are:
The functions to modify the data are:
add (login, from, to , user_name, action_date) rem (login, from, to , user_name, action_date) add_da(login, ue , date, user_name, action_date) rem_da(login, ue , user_name, action_date)
The information about 'Tiers Temps' are stored in a normal table, the columns order must not be modified. There is a line per student.
The information about referents teacher are stored in a normal table named 'orientation_rp'. Each line contains the teacher login, its portals.
The list of students per referent is stored in 'referents_students' table
They are defined in COLUMN_TYPES directory. To add a new column type, the only thing to do is to add 2 new files in COLUMN_TYPES. There is a Python and a JavaScript file per type. Types are defined by a class tree.
The table of types display for each types how to manage the column and the cell. More explanations are in the text.py file.
The server need informations from other servers, in order to be responsive, the requests must be performed in threads. To simplify the program, there is a single thread per data type to be updated.
Some external processes launched by crontab compute some files periodicaly. TOMUSS reread the file is they appear to be modified.
A maximum of work is done by the client, for exemple, average and other computation are not done by the server.
The page is generated by the 'home2' plugin.
Most of the URL are computed with a small javascript function who take the current semester/year from the user selection and append the current ticket.
The page template is stored in 'top2.html' file.
The client load the page that stay open in order to receive updates as JavaScript codes. The client actions are done by inserting image, the image is inserted where the user interaction took place and at the right of the 'cell' tab. It is done so because, an image must stay visible when the user change of page or filter the table. The image is at the same time the server feedback indicating if the action performed well.
The server, indicate that the action was performed with a javascript code. When the client receive this code, it removes the matching images at the right of the 'cell' tab. In the normal case, there is no image at this place.
If an image is not loaded after some time, the image URL is modified in order to retry the aborted load because navigator do not retry failed loads.
If image loads fails and the server is active, then we assume that communication fails because the ticket is no more valid (IP change for example). In this case, the user is asked to authenticate once more.
As TOMUSS started as a trivial program, it was not developed with JavaScript objects. The only objects are:
A Table object should be added in order to clean the code.
These tables are not stored into files. They can be :
The 'Display' framework is currently only used by the 'suivi' page. It allows to add new informations on the 'suivi' page without modifying code source code. Display trees are defined in PLUGINS/suivi_student.py, for example, the information about the 'more_on_suivi' plugin is defined as:
from .. import display # The display framework
def display_more_on_suivi(server):
return configuration.more_on_suivi(server.suivi_login, server)
display.display('MoreOnSuivi', 'BodyRight', 9, data=display_more_on_suivi)
On the javascript side PLUGINS/suivi_student.js:
function DisplayMoreOnSuivi(node)
{
// To use the data from Login display plugin : display_data['Login']
// If 'is_a_teacher' is true, it is the teacher view.
// To hide the block from screen: return ''
return node.data ; // Display the server computed content without change
// It is possible to return values to insert in the container-DIV attributes
// return ["HTML to display", [classes...], [styles...], "other attrs"]
// see DisplayCellBox for an usage example
}
// Defines the data needed from other display plugin.
// The plugin will be called once data is retrieved from server.
// The default value for this function is itself: ['MoreOnSuivi']
// It is possible to wait data from multiple plugins:
// DisplayMoreOnSuivi.need_node = ['MoreOnSuivi', 'Login'] ;
// If no data is needed, use an empty required list: []
On the CSS side PLUGINS/suivi_student.css:
.MoreOnSuivi { background: white }
The node object has following attributes: name, children, containers, priority.
On the javascript side, a display function can call other display function multiple times. For example DisplayUEGrades call CellBox to display each grade.
If the user has not defined its language in its preferences, it is the browser languages that is taken. The server default language is used to created table for column names, comments, value of literals as YES, NO...
The translations are in TRANSLATIONS and LOCAL/LOCAL_TRANSLATIONS directories. The LOCAL translation override the default translation.
some message ID are not in the source code because they are computed. for example for the column types and the attributes.
Message ID format is usualy in this form: TYPE_PLUGIN_ID where TYPE is
TOMUSS retrieves the current student list without knowing to which semester it applies.
The procedure to change of semester is the following:
| When | What |
|---|---|
| You want to use the next semester |
In your local configuration
file, add the new semester with a new port number : suivi.add(2009, 'Automne' , socket.getfqdn() + ':%d', 8891) You can indicate the same port number for multiple semesters, in this case, only one process is launched for these semesters. Set the next semester name in 'year_semester_next' in the TOMUSS configuration table. The tables are editable in both semesters. But current student lists do not appear in the next semester. Il you want to update student lists for the next semester, add it to year_semester_update_student_list in the configuration table. |
| New lists of students are accessible. |
Set 'year_semester' to the same value than 'year_semester_next'. Copy the 'referents_students' table in the new semester. Restart TOMUSS: make stop ; make Il you want to continue update student lists for in the old semester, add it to year_semester_update_student_list in the configuration table. Il you want to allow the previous semester tables to be editable, add it to year_semester_modifiable in the configuration table. |
If the table of the previous semester must be modifiable by default, you can indicate it in the configuration table in the modifiable semester list. The current and next semester are modifiable by default.
Table templates are stored in 'TEMPLATES' directory. When loading a page, if the file match a template it is applied, if the semester match a template it is applied. Only one template can be applied. The templates may defines :
Semester templates defined are:
Table templates defined are:
The table template must be created in order to match the teaching organisation.
Do not store information in the module itself because it will be lost when the TEMPLATE is reloaded (it is triggered by a file change).
The recommended way to define columns in a template is the table method update_columns. It will create and update columns definitions.
If you have configuration parameters associated to your Plugin/Template it is a good idea to use 'utilities.Variables' in order to associate the values to the TOMUSS table '0/Variables/template_name' So the user can change the configuration at running time. You can give access rights to this table to non-root users.
The URLs start by a ticket. If there is no ticket, the navigator is redirected to a service in order to get a ticket. If CAS is not used, then a random ticket number is generated.
The tickets are ``exchanged´´ between the TOMUSS processes as a python module containing all the valid tickets.
To be valid, a ticket must be used with the same navigator and the same IP than the first time.
'ticket.py' define the ticket object that store tickets and parse URLs. It is a generic and not configuration dependant.
A new ticket can be used to revalidate an old ticket, it is useful if the client IP changed.
'authentication.py' can be redefined by a local plugin in order to have the good code for the functions 'ticket_login_name' and 'ticket_ask'. If CAS is not used, then these functions assume that the authentication is done with Apache/.htaccess/.htpasswd with the Basic authentication method (the clear password is checked with su)
In regtest mode, the URLs accept =username as a valid ticket for the user.
If you have multiple suivi servers, then the configuration must be updated when there is a new suivi server. If you have only one suivi server for all the semesters, then no change is needed.
TOMUSS urls are for example:
To have TOMUSS working with nicer URLs we can configure Apache as:
<VirtualHost tomuss.fr:80> ServerName tomuss.univ-lyon1.fr RewriteEngine On # It is the web server (not TOMUSS) that send the static files (UNTESTED) RewriteRule ^/files/([0-9.]+)/(.*) /home/tomuss/TMP/$1/$2 [L] RewriteRule ^(.*) http://tomuss1.fr:8888$1 [P] </VirtualHost> <VirtualHost tomusss.fr:80> ServerName tomusss.univ-lyon1.fr RewriteEngine On RewriteRule ^(.*/2009/Printemps/.*) http://tomuss3.fr:8890$1 [P] RewriteRule ^(.*/2008/Automne/.*) http://tomuss2.fr:8889$1 [P] # 'Suivi' on the current semester RewriteRule ^(.*) http://tomuss2.fr:8889$1 [P] </VirtualHost>
We can configure NGINX with:
proxy_buffering off; # To be really interactive
proxy_read_timeout 1000000; # To keep connection open
server {
listen 80;
server_name tomuss.fr;
# It is the web server (not TOMUSS) that send the static files
location ~ ^/files/ {
root /home/tomuss/TMP ; # Must be readable by NGINX
add_header Cache-Control "max-age=86400";
if ( $request_uri ~* "\.gz$" ) {
add_header Content-Encoding gzip;
add_header Cache-Control "max-age=86400";
}
rewrite ^/files/([0-9.]+)/([^/]*) /$1/$2 break ;
}
location / { proxy_pass http://tomuss1.fr:8888; }
}
server {
listen 80;
server_name tomusss.fr;
location / { proxy_pass http://tomuss3.fr:8890/2009/Printemps/; }
location ~ ^/(=[^/]*/)?2009/Printemps { proxy_pass http://tomuss3.fr:8890; }
location ~ ^/(=[^/]*/)?2008/Automne { proxy_pass http://tomuss2.fr:8889; }
}
With this, the URLs become:
The initial TOMUSS configuration is stored in configuration.py. DB is the production database name.
configuration.py must not be modified because it will be overwrited by each TOMUSS release. The file LOCAL/config.py must be edited to customize and configure TOMUSS.
This customization is done by replacing default functions by yours. For the UCBL university there is:
# The login and the student ID are not the same at the UCBL
# login_to_student_id, the_login, login_to_id (JavaScript)
from . import student_id
# get_ue_dict: Retrieve all the informations about all the UE
from . import spiral
if not regtest:
# ticket_login_name: Get login name from ticket
# ticket_ask: redirect the browser to ask a ticket
from . import auth
# stupid_password: Returns True if the password is stupid
from . import checkpassword
# students: Iterator on all the students of an UE, returns :
# (student_id, firstname, surname, mail, group, sequence)
from . import students_of_ue
Most of the configuration values are modifiable while TOMUSS is running by editing as 'root' the table named 'http://......../0/Dossiers/config_table' The values in this table have precedence over values stored in the configuration source files. Nevertheless the first time, this table is created using the configuration source files
The home page of root user contains a link on the plugin configuration table. For each plugin, you can indicate a list of allowed item :
It is recommended to use group name because most of the time, a few plugins have the same users allowed. The user groups are defined in the config ACLS table. On the first column you enter an item as in the plugin table, on the second column, you indicate to which group you add the item.
If a group contains items prefixed by '!', then they evaluated first, and if the user is contained in the group, it will be rejected. There is an exception: if the group or list contains only one '!' item, then it returns True if the user is not in the group.
Example displayed as a tree:
staff
* ldap:GROUP_X # Evaluated after !, so some user may be rejected
* !FORBIDEN # Define the list of forbidden users
* ldap:GROUP_Y
* !USER_A # It will not be in FORBIDEN
* !grp:teachers # No teachers in FORBIDEN
* teachers
* ldap:GROUP_T
* USER_B
* administratives # FORBIDEN is tested before administrative
* ldap:GROUP_A # If users are in GROUP_Y they will not
* USER_C # be in 'administrative' group (even USER_C)
The makefile goal regtest run an infinite loop on some server tests. The loop is broken if there is a problem.
The URL http://SERVER/2009/Test/javascript_regtest_ue run some javascripts tests on the user interface. These tests works on FireFox and Opera but not on IE.
The makefile goals start and stop allow to manage the TOMUSS services. There is one database modification service and one 'suivi' service per semester in order to not have huge processes. The 'suivi' processes are huge because they load the full semester in memory.
UPDATE SINCE Version 5.2.0: once students are indexed by SCRIPTS/bilan.py the 'suivi' process do not load complete semester but only needed tables. In this case only one 'suivi' process is needed. To use only one process, use the same port for all the semesters.
The goal install of the Makefile runs the script install that do all the work to replace a running TOMUSS by the new release. IT IS A VERY BAD IDEA TO RUN THIS SCRIPT WITHOUT UNDERSTANDING IT.
Two symbolic links (DB and BACKUP_DB) points on directory where the database is stored.
Required packages : python-ldap, python-imaging, gzip, inkscape, gettext
Recommended packages : gnuplot, graphviz, rsync (distant mirroring), catdoc (csv extract from xls)
The TOMUSS root can use the following features:
Some non automatic work to do: see LOCAL/Makefile
When loading a Python module, never store the 'configuration' values in local variables because the configuration module may not be fully loaded.
Use the function 'unload_module' to unload a Python module in order to not have a memory leak.
Never modify the database files if the table is loaded in the TOMUSS server.
If you want your TOMUSS customization to not be destroyed by a version change you must follow the procedure. The functions listed are the ones that you need to modify, but you can modify any function you want.
In the following table the javascript functions must not be modified in the javascript source, they can be redefined by LOCAL/config.py script using this procedure: import files
files.files['lib.js'].append("a_key", """
function the_function_to_be_redefined()
{
}
"""
| The Python function must not be modified in the Python sources, they must be redefined in LOCAL/config.py script using this procedure: import a_module old_one = a_module.to_be_redefined def to_be_redefined(): old_one() a_module.to_be_redefined = to_be_redefined |