Depo - Data Management

Initial transfer protocol

The initial program had the file paths stored within the source code and the program was called from a local PC using a batch file which linked to a remote server. Users had to define themselves via an individual username to provide the correct path connections for their machine. This was due to different machines having different mapped drives and devices therefore different drive letters. The file structure maintained the same system used when manual movement of data was employed.

This approach was basic and meant the source code needed recompiling every time a new user or machine was added to the Depo system and although the year code suffix was used data from all years where stored in the same root directory under the supervisor initials.

File Management

This was amended in the current version of Depo with a massive rewrite of the source code and a new approach to the file management and transfer protocols. Now all data would be split into those files just used for publication (www) and those files used for structure solution and processing (data).tree

This approach meant that archiving of data could be done much more efficiently by allowing the data part of the server which is memory intensive to be copied then removed whilst maintaining the www information which would always be online via the webserver and considerably smaller than the data part. Also archiving the www part of the server would be faster as only the essential files would be present.

This split in data handling meant that a further step could be installed into the file transfer, this was the compression of the raw data and processing files into a single zip file further reducing memory usage.

tree

The data management was further improved by taking advantage of the redundant year suffix and adding a new layer into the file hierarchy. This reduced clutter of data and increased ease of access.

By using this extra layer in both the data and www parts of the server older years could be removed preferentially when archiving and aided in backing-up and storage management.

The existing source code will allow up to 26 years worth of data to be stored using single letter suffixes by which point I would hope someone else has come up with something better!

Path Management

With the increased complexity in file structure and the need to alter the source code to implement it further alterations were made to the file management part of the program by modifying the path handling protocols. The use of paths written into the source code was as mentioned above most undesirable.

Depo would still be hosted on a remote server along with all the files needed to compress the x-ray data and database files but when Depo was run on a new system it would look for and create if not present a temporary folder to increase file maniplication speeds and a configuration file. The configuration file contains a set of default paths and instructions when created and the user alters them for their machine. This removed user IDs and the need to recompile Depo for new users and machines.

*** Please delete this line. If these values do no work please edit depo.conf and change the default paths ***
edit this to just the path Line 1: Default path to the server data :g:\data
edit this to just the path Line 2: Default path to the server www :g:\www
edit this to just the path Line 3: Default path to the database :g:\info\database.txt
edit this to just the path Line 4: Default path to ortep.exe :c:\wingx\ortep32\ortep32.exe

Furthermore this new system would also allow UNC format paths and removed the need to map network drives:

\\xray\......\data
\\xray\.......\www
\\xray\......\database.txt
c:\wingx\ortep32\ortep32.exe

07/07/2010