source: 1dwg/trunk/doc/source/overview.rst @ 250

Last change on this file since 250 was 250, checked in by prjemian, 8 years ago

refs #24

  • Property svn:eol-style set to native
  • Property svn:keywords set to Author Date Id Revision URL
File size: 18.4 KB
RevLine 
[241]1.. $Id$
2
3=================
4Overview
5=================
6
7.. index:: canSAS; objective
8.. index:: canSAS; aims
9.. index:: ! canSAS
10.. index:: ! I(Q)
11
12One of the first aims of the **canSAS**
13(Collective Action for Nomadic Small-Angle Scatterers)
14forum of users, software developers, and facility staff was to discuss
15better sharing of SAS data analysis software. The canSAS forum
16(http://www.smallangles.net/canSAS)
17identified that a significant need within the SAS
18community can be satisfied by a robust, self-describing, text-based, standard format to
19communicate reduced one-dimensional small-angle scattering data, :math:`I(Q)`, between users
20of our facilities. Our goal has been to define such a format that leaves the data file
21instantly human-readable, editable in the simplest of editors, and importable by simple
22text import filters in programs that need not recognise advanced structure in the file
23nor require advanced programming interfaces. The file should contain both the primary
24data of :math:`I(Q)`
25and also any other descriptive information (:index:`metadata`)
26about the sample, measurement, instrument, processing, or analysis steps.
27
28Objective
29================
30
31.. index:: ! cansas1d/1.1 standard
32.. index:: canSAS; objective
33
34The cansas1d/1.1
35standard meets the objectives for a 1D standard, incorporating :index:`metadata`
36about the measurement, parameters and results of processing or analysis steps.
37Even multiple measurements (related or unrelated) may be included within a single XML
38file.
39
40
41General Layout of the XML Data
42====================================
43
44.. index:: I(Q)
45
46The canSAS 1-D standard for reduced 1-D SAS data is implemented using XML files. A
47single file can contain SAS data from a single experiment or multiple experiments. All
48types of relevant data (:math:`I(Q)`, :index:`metadata`)
49are described for each experiment. More details are provided below.
50
51.. index:: element; SASroot
52.. index:: element; SASentry
53
54The basic elements of the cansas1d/1.1 standard are shown in the following table.
55After an XML header, the root element of the file is :ref:`SASroot`
56which contains one or more :ref:`SASentry`
57elements, each of which
58describes a single experiment (data set, time-slice, step in a series, new sample,
59etc.). Details of the *SASentry* element are also shown in the
60next figure.
61See the section :ref:`elements`
62for examples of cansas1d/1.1 XML data files.
63Examples, Case Studies, and other background information
64are below. More discussion can be found on the
65canSAS 1D Data Formats Working Group page
66(http://www.smallangles.net/wgwiki/index.php/1D_Data_Formats_Working_Group)
67and its discussion page. 
68(http://www.smallangles.net/wgwiki/index.php/Talk:1D_Data_Formats_Working_Group)
69
70        .. figure:: ../../graphics/10-minimum.png
71            :alt: cansas1d/1.1 standard block diagram, minimum elements
72           
73            block diagram of minimum elements required for *cansas1d/1.1* standard
74
75:ref:`SASroot`
76        the root element of the file (after the XML header)
77
78:ref:`SASentry`
79        describes a single experiment (data set, time-slice, step in a series, new sample, etc.)
80
81.. index:: ! XML header
82
83.. _XML.header:
84
85.. rubric:: Required header for cansas1d/1.1 XML files
86
87.. code-block:: xml
88        :linenos:
89       
90        <?xml version="1.0"?>
91        <SASroot version="1.1"
92                xmlns="cansas1d/1.1"
93                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
94                xsi:schemaLocation="cansas1d/1.1 http://svn.smallangles.net/svn/canSAS/1dwg/trunk/cansas1d.xsd"
95                >
96
97.. index: element; SASroot
98.. index: element; SASentry
99.. index: element; SASdata
100.. index: element; Idata
101.. index: element; SAStransmission_spectrum
102.. index: element; Tdata
103.. index: element; {any}
104.. index: element; SASsample
105.. index: element; SASinstrument
106.. index: element; SASsource
107.. index: element; SAScollimation
108.. index: element; SASdetector
109.. index: element; SASprocess
110.. index: element; SASnote
111
112.. rubric:: Basic elements of the canSAS 1-D standard
113
114===============================   ===========================================================================
115Element                           Description
116===============================   ===========================================================================
117:ref:`XML Header<XML.header>`     descriptive info required at the start of every XML file
118:ref:`SASroot`                    root element of XML file
119:ref:`SASentry`                   data set, time-slice, step in a series, new sample, etc.
120:index:`Title`                    for this particular :ref:`SASentry`
121:index:`Run`                      run number or ID number of experiment
122:ref:`{any}`                      any XML element can be used at this point
123:ref:`SASdata`                    this is where the reduced 1-D SAS data is stored
124:ref:`Idata`                      a single data point of :math:`I(Q)` (and related items) in the dataset
125:ref:`SAStransmission_spectrum`   any transmission spectra may be stored here
126:ref:`Tdata`                      a single data point in the transmission spectrum
127:ref:`{any}`                      any XML element can be used at this point
128:ref:`SASsample`                  description of the sample
129:ref:`SASinstrument`              description of the instrument
130:ref:`SASsource`                  description of the source
131:ref:`SAScollimation`             description of the collimation
132:ref:`SASdetector`                description of the detector
133:ref:`SASprocess`                 description of each processing or analysis step
134:ref:`SASnote`                    anything at all
135===============================   ===========================================================================
136
137
138Rules
139========================
140
141.. index:: ! unit
142.. index:: ! Q
143.. index::
144        single: validation; against XML Schema
145        single: geometry; Q
146        see: units; unit
147        single: geometry; translation
148        single: geometry; orientation (rotation)
149
150#. A cansas1d/1.1 XML data files will adhere to the standard if it can
151        successfully :ref:`validate` against the established XML Schema.
152        (http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/cansas1d.xsd)
153#. :math:`Q=(4 \pi / \lambda) \sin(\theta)`
154        where :math:`\lambda` is the wavelength of the radiation,
155        and :math:`2\theta` is the angle through which the detected radiation has been scattered.
156       
157        .. figure:: ../../graphics/Q-geometry.jpg
158            :alt: Q geometry
159            :height: 200 px
160           
161            definition of Q geometry for small-angle scattering
162
163#. units to be given in standard SI abbreviations (eg, m, cm, mm, nm, K)
164        with the following exceptions:
165       
166        ============   ===================================================
167        use this       to mean this
168        ============   ===================================================
169        um             micrometres
170        C              celsius
171        A              Angstroms
172        percent        %
173        fraction       fraction
174        a.u.           arbitrary units
175        none           no units are relevant (such as dimensionless)
176        ============   ===================================================
177
178#. where reciprocal units need to be quoted, the format shall be "1/abbreviation",
179        such as ``1/A``
180#. use ``^`` to indicate an exponent (rather than ``**``), such as ``m^2``
181#. when raised to a power, use similar to ``A^3`` or ``1/m^4`` 
182        (and not ``A3`` or ``A**3`` or ``m-4``)
183#. :index:`coordinate axes`:
184        (See the :ref:`compatibility` section)
185       
186        a. :math:`z` is along the flight path (positive value in the direction of the detector)
187        #. :math:`x` is orthogonal to :math:`z` in the horizontal plane (positive values
188                increase to the right when viewed towards the incoming
189                radiation)
190        #. :math:`y` is orthogonal to :math:`z` and :math:`x` in the vertical plane
191                (positive values increase upwards)
192       
193        .. figure:: ../../graphics/translation-orientation-geometry.jpg
194            :alt: coordinate axes as viewed from the source
195            :height: 200 px
196           
197            definition of translation and orientation geometry as viewed from the source towards the detector
198       
199        .. figure:: ../../graphics/translation-orientation-geometry-2.jpg
200            :alt: coordinate axes as viewed from the detector
201            :width: 200 px
202           
203            definition of translation and orientation geometry as viewed from the detector towards the source
204
205#. orientation (angles) describes single-axis rotations (rotations about
206        multiple axes require more information):
207       
208        a. :index:`roll` is about :math:`z`
209        #. :index:`pitch` is about  :math:`x`
210        #. :index:`yaw` is about  :math:`y`
211
212#. Binary data is not supported
213
214
[250]215.. _XmlWriter:
[241]216
[250]217Converting data into the XML format: *XmlWriter*
218====================================================
219
[241]220.. index:: ! xmlWriter
221
222The canSAS/xmlWriter (http://www.smallangles.net/canSAS/xmlWriter/)
223is a WWW form
224to translate three-column ASCII text data into the cansas1d/1.1 XML
225format. This form will help you in creating an XML file with all the required
226elements in the correct places. The form requests the SAS data of *Q*, *I*, and *Idev*
227(defined elsewhere on this page) and some basic :index:`metadata`
228(title, run, sample info, ...).
229
230Press the *Submit* button and you will receive a nicely
231formatted WWW page with the SAS data. If you then choose *View page source*
232(from one of your browser menus), you will see the raw XML of the cansas1d/1.1 XML format
233and you can copy/paste this into an XML file.
234
235The SAS data that you paste into the form box is likely to be copied directly from
236a 3-column ASCII file from a text editor. Line breaks are OK, they will be treated
237as white-space as will tabs and commas. Do not be concerned that the data looks
238awful in the form entry box, just check the result to see that it comes out
239OK.
240
241
242Documentation and Definitions
243========================================
244
[250]245.. _cansas1d.xsd:
246
[241]247XML Schema
248-------------
249
[250]250The *cansas1d.xsd* :index:`XML Schema` 
251(http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/cansas1d.xsd)
[241]252defines the rules for the XML file format and is used to
253validate any XML file for adherence to the format.
254
255        TRAC (view source code highlighted by bug tracking system)
256                http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/cansas1d.xsd
257
258        SVN (view raw source code from version control system)
259                http://svn.smallangles.net/svn/canSAS/1dwg/trunk/cansas1d.xsd
260
261XML stylesheets
262----------------
263
264An :index:`XML stylesheet`, or *XSLT* (http://www.w3schools.com/xsl/),
265can be used to extract :index:`metadata` 
266or to convert into another file format. The
267default canSAS stylesheet *cansasxml-html.xsl*
268(http://svn.smallangles.net/svn/canSAS/1dwg/trunk/cansasxml-html.xsl)
269should be copied into each folder with canSAS XML data
270file(s). It can be used to display the data in a supporting WWW browser
271(such as Firefox or Internet Explorer) or to import into Microsoft Excel
272(with the added XML support in Excel).
273
274.. tip:: See the excellent write-up by Steve King, ISIS,
275        (http://www.isis.rl.ac.uk/archive/LargeScale/LOQ/xml/cansas_xml_format.pdf)
276        for an example.
277
278By default, MS Windows binds *.xml* files to start
279Internet Explorer. Double-clicking on a canSAS XML data file with the
280*cansasxml-html.xsl* (see above tip)
281stylesheet in the same directory will produce a
282WWW page with the SAS data and selected metadata.
283
284
285Suggestions for support software that write cansas1d/1.1 XML data files
286-------------------------------------------------------------------------
287
288.. index::
289        single: file; Writing cansas1d/1.1 files
290        single: best practices
291
292Some common best practices have been identified in the list below.
293
294* be sure to update to the latest SVN repository revision:
295
296        .. code-block:: text
297
298                svn update
299
300* check the output directory to see if it contains the default XSLT file
301* copy the latest XSLT file to the output directory if either:
302        * the output directory contains an older revision
303        * the output directory does not have the default XSLT file
304* The most recent XSLT file can be identified by examining the file
305        for the *$ Revision: $* string, such as in the next example.
306       
307        .. code-block:: text
308       
309                # $Revision$
310
311
312Examples and Case Studies
[242]313=========================================================
[241]314
315.. index:: XML file; cansas1d.xml
316
317**Basic example** 
318        http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/cansas1d.xml
319       
320    Note that, for clarity, only one row of data is
321    shown. This is probably a very good example to use as a starting point for
322    creating XML files with a text editor.
323
324.. index:: XML file; bimodal-test1.xml
325.. index:: case study; bimodal test data
326
327**Bimodal test data**
328        http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/bimodal-test1.xml
329
330    Simulated SAS data (with added noise) calculated from model bimodal size
331    distribution to test size distribution analysis routines.
332
333
334.. index:: case study; glassy carbon round robin
335
336**Glassy Carbon Round Robin**
337        http://www.smallangles.net/wgwiki/index.php/Glassy_Carbon_Round_Robin
338
339    Samples of a commercial glassy carbon
340    measured at several facilities worldwide.
341
[242]342.. index:: case study; dry chick collagen SAXS
[241]343
[242]344**dry chick collagen SAXS**
345        see :ref:`case_study-collagen` section
346       
347        SAXS data from *dry chick collagen* illustrates the
348        minimum information necessary to meet the requirements of the standard
349        format.
[241]350
[242]351.. index:: case study; AF1410 steel SANS
[241]352
[242]353**AF1410 steel SANS**
354        see :ref:`case_study-af1410` section
355       
356        SANS data from *AF1410 steel* using magnetic
357        contrast variation (with multiple samples and multiple data sets for each
358        sample), the files can be viewed from the TRAC site (no description yet).
359       
360        http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/examples/af1410/
[241]361
[242]362.. index:: XML file; cansas1d-template.xml
363
[244]364**Test all the cansas1d/1.1 rules**
[242]365        http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/cansas1d-template.xml
366       
[244]367    The *cansas1d-template.xml* data file is used to test all the rules in the XML
[242]368    Schema. This is probably not a very good example to use as a starting point
369    for creating XML files with a text editor since it tests many of the
370    special-case rules.
371
372
373XML layout for multiple experiments
374-------------------------------------------
375
376.. index::
377        single: multiple experiments
378        single: multiple data sets
379
380Each experiment is described with a single *SASentry* element. The
381fragment below shows how multiple experiments with multiple data sets
382can be included in a single XML file. This illustrates using more than
383one *SASentry* and more than one *SASdata* element.
384
385.. literalinclude:: ../src/brief-sketch-multiple.xml
386   :language: xml
387   :linenos:
388
389Full examples of canSAS XML files with multiple experiments
390include:
391
392.. index:: XML file; W1W2.XML
393
394**multiple data sets**
395        ISIS LOQ SANS instrument:
396        http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/W1W2.XML
397
398.. index:: XML file; cs_af1410.xml
399
400**multiple samples, multiple data sets**
401        AF1410 steel SANS contrast variation study from NIST:
402        http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/examples/af1410/cs_af1410.xml
403       
404    SANS study using magnetic
405    contrast variation (with multiple samples and multiple data sets for each
406    sample), the files can be viewed from the TRAC site (no description yet).
407
408Foreign Elements
409------------------------
410
411.. index:: XML; foreign elements
412
413To allow for inclusion of elements that are not defined by the
414*cansas1d.xsd* XML Schema, XML **foreign elements** 
415are permitted at select locations in the
416cansas1d/1.1 format. Please refer to the :ref:`XML Help` section
417for more help with XML foreign elements.
418
419.. note::  **Need to make another example.**
420        This example was based on v1.0.  With v1.1, there is no need for
421        the foreign namespace in this example. 
422
423There is an example that demonstrates the use of a foreign namespace:
424http://svn.smallangles.net/trac/canSAS/browser/1dwg/data/Glassy%20Carbon/ISIS/GLASSYC_C4G8G9_withTL.xml
425
426This example uses a foreign namespace to record the transmission spectrum related to
427the acquisition of the SANS data at a time-of-flight facility. Look near line 153
428for this element::
429
430        <transmission_spectrum xmlns="urn:transmission:spectrum">
431
432The foreign namespace given (``urn:transmission:spectrum``) becomes the
433default namespace for just the *transmission_spectrum* element.*
434
435Also refer to canSAS TRAC ticket #47
436(http://svn.smallangles.net/trac/canSAS/changeset/47)
437for an example of arranging the content in
438*SASprocessnote* to avoid the use of foreign namespace
439elements.
440
441Support tools for Visualization & Analysis software
442=========================================================
443
444Support for importing cansas1d/1.1 files exists for
445these languages and environments:
446
447.. index::
448        single: binding; FORTRAN
449        see: FORTRAN; binding, FORTRAN
450
451**FORTRAN**
452        See the :ref:`fortran.binding` section.
453
454.. index::
455        single: binding; IgorPro
456        see: IgorPro; binding, IgorPro
457
458**IgorPro**
459        See the :ref:`IgorPro.binding` section.
460
461.. index::
462        single: binding; Java JAXB
463        see: Java JAXB; binding, Java JAXB
464
465**Java JAXB**
466        See the :ref:`Java.JAXB.binding` section.
467
468.. index::
469        single: binding; Microsoft Excel
470        see: Microsoft Excel; binding, Microsoft Excel
471
472**Microsoft Excel**
473        Support for Microsoft Excel is provided through the default canSAS stylesheet,
474        *cansasxml-html.xsl* (http://svn.smallangles.net/svn/canSAS/1dwg/trunk/cansasxml-html.xsl).
475        The ISIS **LOQ** instrument (http://www.isis.stfc.ac.uk/instruments/loq/loq2470.html)
476        has provided an excellent description of how to import data from the cansas1d/1.1 format into Excel.
477        Also note that the old WWW site (http://www.isis.rl.ac.uk/archive/LargeScale/LOQ/loq.htm)
478        may still be available.
479
480.. index::
481        single: binding; PHP
482        see: PHP; binding, PHP
483        single: xmlWriter
484
485**PHP**
486        See the :ref:`PHP.binding` section.
487       
488        The *canSAS/xmlWriter* (http://www.smallangles.net/canSAS/xmlWriter/)
489        is implemented in PHP (http://www.php.net) and writes a cansas1d/1.1 data
490        file given three-column ASCII data as input.  The code uses
491        *DomDocument* (http://www.php.net/DomDocument)
492        to build the XML file.  Look for the line beginning with::
493       
494                function prepare_cansasxml($post)
495       
496        Another example of *DomDocument*
497        is in the ``function surveillance($post)`` where
498        logging information is inserted into an XML file.
499       
500        PHP source: http://svn.smallangles.net/trac/canSAS/browser/1dwg/trunk/php/xmlWriter/index.php
501
502.. index::
503        single: binding; Python
504        see: Python; binding, Python
505        single: xmlWriter
506
507**Python**
508        See the :ref:`Python.binding` section.
509
510.. index::
511        single: binding; XML Stylesheet (XSLT)
512
513**XSLT**
514        *XSLT*  (useful in a web browser) is described later in the
515        :ref:`Example.XML.Stylesheets` section.
516
517
518Software repositories (for cansas1d/1.1 standard)
519=========================================================
520
521TRAC (bug reporting)
522        http://svn.smallangles.net/trac/canSAS/browser/1dwg/tags/v1.1
523
524SVN (*subversion* revision control system)
525        http://svn.smallangles.net/svn/canSAS/1dwg/tags/v1.1
Note: See TracBrowser for help on using the repository browser.