@page title="Data Loaders" keywords="data loader"> <@sect title="Introduction">
Data loaders are Java objects that load data from certain type of data source, such as a CSV file or MySQL data-base, and expose the data as variables for the templates. For example, a data loader was invoked in the <@a href="qtour.html">Quick Tour@a> when you have written <@c>csv(data/birds.csv)@>.
Data loaders are typically invoked:
Data loaders that load data directly from files will want you to give the path of the file as parameter. There you have to give <@a href="overview.html#realPath">real path@a>. If you give a relative path, then it will be interpreted relatively to the <@s>dataRoot@s>, which defaults to the <@s>sourceRoot@s> if you didn't specified it. The data files can be outside the <@s>dataRoot@s> directory; it is used only as a base directory.
<@note>I have not written a database/SQL/JDBC data loader yet... Of coruse, you can write a such data loader yourself. Contributions are welcome!@note> <@dataLoader name="csv"> <@param name="path" type="string">The path of the CSV file.@param> <@param name="options" type="hash" optional=true>The list of valid options:Examples (with TDD syntax):
The <@c>csv@c> data loader parses CSV (Column Separated Values) file, or other file of similar formats (as tab divided text, comma separated values), and returns it as a sequence. The sequence is the list of the table rows, and each row is a hash, where you can access the cells with the column name. The column names (headers) are stored in the first row of the CSV file (unless you use the <@c>headers@c> option), so this row is not part of the result sequence.
For example, if this is the CSV file:
<@prg> name;color;price rose;red;10 rose;yellow;15 tulip;white;6 @prg>and you load it into the variable <@c>flowers@c>, then you can print it as:
<@prg><#noparse>${flower.name} | ${flower.color} | ${flower.price} #list> |
and the output will be:
<@prg><#noparse>rose | red | 10 |
rose | yellow | 15 |
tulip | white | 6 |
The rows are not only hashes, but also sequences, so instead of the column name you can use the 0 based column index. Thus the above template could be written like this as well, just it is less readable:
<@prg><#noparse>${flower[0]} | ${flower[1]} | ${flower[2]} #list> |
and actually then it would be simpler to write:
<@prg><#noparse>${cell}#list> #list> |
The values (cells) in the CSV file will be always exposed as string variables, unless you specify a different type in the header cells directly. To specify the type, type a colon after the column name, and a type identifier. The type identifier can be: <@c>n@c> (or <@c>number@c>) or <@c>b@c> (or <@c>boolean@c>) or <@c>d@c> (or <@c>date@c>) or <@c>t@c> (or <@c>time@c>) or <@c>dt@c> (or <@c>dateTime@c>) or <@c>s@c> (or <@c>string@c>). For example, if you want to expose the price column as number, and not as string (so you can do arithmetic with it, or use the number formatting facilities), then the CSV file would be:
<@prg> name;color;price:n rose;red;10 rose;yellow;15 tulip;white;6 @prg>Numerical values must use dot (<@c>.@c>) as decimal separator, and no grouping, unless you change these with the <@c>decimalSeparator@c> and/or <@c>groupingSeparator@c> options.
Boolean values must be one of <@c>true@c>, <@c>yes@c>, <@c>y@c>, <@c>1@c>, <@c>false@c>, <@c>no@c>, <@c>n@c>, <@c>0@c>, or the words defined be the <@c>altTrue@c> and <@c>altFalse options@c>. Upper- and lower-case letters are not distinguished.
Date, time and date-time values use common SQL format with optional time zone, as <@c>"2003-06-25"@c>, <@c>"22:30:08"@c>, <@c>"2003-06-25 10:30:08 PM"@c>, <@c>"2003-06-25 10:30:08 PM GMT+02:00"@c>. But if you use option <@c>dateFormat@c>, <@c>timeFormat@c>, or <@c>datetimeFormat@c>, then that format has to be used for dates, times, and date-times respectively. If the time zone is not given in a value, the value of the <@s>timeZone@s> setting is used.
The variable returned by the <@c>csv@c> data loader is not only a sequence, but also a hash at the same time, that contains one key: <@c>headers@c>. This is a sequence that stores the column names.
<@prg><#noparse> <#list flowers.headers as h> - ${h} #list> #noparse>@prg>will print:
<@prg> - name - color - price @prg>Note that only the name of the column is returned, not the type identifier stuff (as <@c>:n@c>).
@dataLoader> <@dataLoader name="json"> <@param name="path" type="string">The path of the JSON file.@param> <@param name="charset" type="string" optional=true>The charset used for reading the JSON file. Defaults to the <@s>sourceEncoding@> setting.@param>Examples (with TDD syntax):
This data loader parses a JSON file. The JSON file must contain a single top-level JSON value of any type, like a JSON array, or a JSON object, or a JSON string, etc. Example file content (this contains a JSON array on the top-level):
<@prg> [ { "name": "Jean Test", "maidenName": "Jean Test", "age": 20, "skills": [ "HTML", "CSS" ], "testResults": { "a": 10.5, "b": 20, "c": 30 }, "decided": true }, { "name": "José Test", "maidenName": null, "age": 30, "skills": [ "Ruby", "C++", "Cuda" ], "testResults": { "a": 20, "b": 30, "c": 40 }, "decided": false } ] @prg>Let's loaded the above JSON array into the <@c>applicants@c> data-model variable:
<@prg> data: { ... applicants: json(applicants.json) ... } @prg> <@note>Above, the <@c>applicants:@c> label is required because this file defines a JSON array, so we need a name with which we can refer to it from the templates. If a JSON file defines a JSON object instead (like <@c>{ "applicants": [ <@r>...@r> ], "otherStuff": <@r>...@r> }@c>), then, if you omit the label, the top-level name-value pairs from the JSON object will be added to the data-model directly.@note>Now you could print the name and age of all applicants, and print their skills in nested listing like this:
<@prg><#noparse> <#list applicants as applicant> ${applicant.name} (age: ${applicant.age}) <#list applicant.skills as skill> - ${skill} #list> #list> #noparse>@prg>That is, JSON arrays will be FTL (FreeMarker Template Language) sequences, JSON objects will be FTL hashes, JSON strings will be FTL strings, JSON numbers will be FTL numbers, JSON boolean will be FTL boolean. JSON null-s will be FTL undefined variables (Java <@c>null@c>-s), so for <@c>applicants[1].maindenName@c> FreeMarker would give an undefined variable error, and you need to write something like <@c>applicants[1].maindenName!'N/A'@c> instead.
The loaded JSON will also act as an FTL node tree (an advanced FreeMarker feature), similarly to an XML. This means, among others, that you can get the parent of a value. The parent of a value is its containing array or object:
<@prg><#noparse> <#-- We just pick some skill for this example: --> <#assign skill = applicants[0].skills[0]> <#-- Now let's say we only have the skill variable, and we don't know how was it get. We can still get whose skill it was: --> ${skill} is the skill of ${skill?parent?parent.name}, <#-- skill?parent is the array, the parent of that is the JSON object --> #noparse>@prg>More details about the node nature:
See <@fma>the FreeMarker Manual@fma> for more about nodes and the related directives and built-ins!
@dataLoader> <@dataLoader name="text"> <@param name="path" type="string">The path of the text file.@param> <@param name="charset" type="string" optional=true>The charset used for reading text file. Defaults to the <@s>sourceEncoding@> setting.@param>Examples (with TDD syntax):
This data loader loads a plain text file, and returns that as a string.
@dataLoader> <@dataLoader name="slicedText"> <@param name="path" type="string">The path of the text file.@param> <@param name="options" type="hash" optional=true>The list of valid options:Examples (with TDD syntax):
This data loader loads a text file and slices it to items (strings) by cutting out all occurences of the separator string specified with the <@c>separator@c> option. The result is a sequence of strings.
For example, if this is the <@c>data/stuff.txt@c> file:
<@prg> This is the first item. Still the first item... --8<-- The items are separated with this: --8<-- just don't forget to surround it with empty lines. --8<-- This is the last item. @prg>and it's loaded in the configuration file like: <@prg> data: { stuff: slicedText(data/stuff.txt, {separator:"\n\n--8<--\n\n"}) } @prg><#-- " -->
then the output of this template file
<@prg><#noparse> <#escape x as x?html> <#list stuff as i>${i}#list> #escape> #noparse>@prg>
will be this:
<@prg>This is the first item. Still the first item...
The items are separated with this: --8<-- just don't forget to surround it with empty lines.
This is the last item.@prg>
Note the double <@c>\n@c>-s in the separator that causes the data loader to treat <@c>"–8<--"@c> as separator only if it is surrounded with empty lines.
@dataLoader> <@dataLoader name="tdd"> <@param name="path" type="string">The path of the TDD file.@param> <@param name="charset" type="string" optional=true>The charset used for reading TDD file. Defaults to the <@s>sourceEncoding@> setting.@param>Examples (with TDD syntax):
This data loader parses a <@a href="tdd.html">TDD@a> file. The loaded TDD is interpreted in <@a href="tdd.html#modes">hash mode@a>, and TDD functions will invoke data loaders, exactly like with the <@s>data@s> setting.
See <@example 'tdd' /> for a concrete example.
@dataLoader> <@dataLoader name="tddSequence"> <@param name="path" type="string">The path of the TDD file.@param> <@param name="charset" type="string" optional=true>The charset used for reading TDD file. Defaults to the <@s>sourceEncoding@> setting.@param>This is like the <@a href="#key_tdd"><@c>tdd@c> dataloader@a>, except that it interprets the file as a <@a href="tdd.html">TDD@a> sequence (TDD "sequence mode"), rather than as a TDD hash. So the result will be a sequence (a list), not a hash.
@dataLoader> <@dataLoader name="properties"> <@param name="path" type="string">The path of the "properties" file.@param>Loads a <@a href="properties.html">"properties" file@a>. The result is a hash.
@dataLoader> <@dataLoader name="xml"> <@param name="path" type="string">The path of the XML file.@param> <@param name="options" type="hash" optional=true>Options. See later...@param> <@note>Sometimes XML files are rather source files (as the HTML-s and the JPG in the <@a href="qtour.html">Quick Tour@a>) than data files (as <@c>data/birds.tdd@c> in the Quick Tour), just you need to invoke a template to render them to their final form (e.g. to HTML pages). If this is the case, you should use <@c>renderXml@c> processing mode (<@a href="settings.html#key_modes">see here...@a>), not XML data loader. An example that uses this approach is <@example 'xml_rendering' />.@note>Loads an XML file. This uses the built-in XML wrapper of FreeMarker 2.3 and later; please read <@fma href="xgui.html">FreeMarker Manual/XML Processing Guide@fma> for more information about the usage of the returned variable.
Notes:
The example <@example 'xml_try'/> is a good tool to understand this data loader and its options.
Options:
Evaluates a BeanShell expression, or runs a more complex BeanShell script and uses its <@c>return@c>-ed value.
You can use the predefined variable <@c>engine@c> to access the current FMPP engine instance.
Examples (with TDD syntax):
The returned hash contains custom directives that help HTML template development.
Currently it contains only 1 directive: <@c>img@c>. This is used as HTML <@c>img@c> element, but the <@c>width@c> and <@c>height@c> attributes are calculated automatically (unless these attributes are specified in the template). Also, the processing will stop with error, if the image file pointed by the <@c>src@c> attribute is missing.
Example: If you have loaded the hash into the <@c>html@c> variable, then you can do this in a template:
<@prg><#noparse> <@html.img src="falcon.png" alt="Falcon" /> #noparse>@prg>and the output will be something like this (depending on the concrete dimensions of the image):
<@prg><#noparse>See also the example in <@example 'img_dims' />.
Directive <@c>img@c> accepts any extra parameters (attributes). The extra parameters will be printed as the attributes of the HTML <@c>img@c> element. @dataLoader> <@dataLoader name="xhtmlUtils">
This is the same as <@c>htmlUtils@c>, but it is for XHTML output.
@dataLoader> <@dataLoader name="now"> <@param name="options" type="hash" optional=true>The supported options are:Note: FreeMarker now has a <@c>.now@c> variable, so you don't need this data loader any more. You can print the current date and time with <@c>${'$'}{.now}@c> (or <@c>${'$'}{.now?date}@c> or <@c>${'$'}{.now?time}@c>), and the format and zone is specified by the global <@s>datetimeFormat@s> (or <@s>dateFormat@s> or <@s>timeFormat@s>) and <@s>timeZone@s> FMPP settings.
Examples (with TDD syntax):
This data loader loads the current date and time from the system clock, and produces a string from it. Since FreeMarker has introduced native date/time value support, it is maybe better to use <@a href="pphash.html#key_sessionStart"><@c>pp.sessionStart@c>@a> or <@a href="pphash.html#key_now"><@c>pp.now@c>@a> instead of this data loader.
@dataLoader> <@dataLoader name="get"> <@param name="varName" type="string">The name of the variable.@param> <@param name="subVarName" type="string" optional=true>The name of the sub-variable.@param> <@param name="subSubVarName" type="string" optional=true>...etc. You can specify any number of parameters.@param>This data loader returns the value of a variable that was already put into the <@s>data@s> hash. For example:
<@prg> data: { a: 123 b: get(a) } @prg>Here the value of <@c>b@c> will be the same as the value of <@c>a@c>, which is <@c>123@c>. You can retrieve the value of subvariables with additional parameters:
<@prg> data: { a: { x: 123 y: { q:234 } } b: get(a, x) c: get(a, y, q) } @prg>Here <@c>b@c> will be <@c>123@c>, <@c>c@c> will be <@c>234@c>.
The <@c>get@c> data loader was introduced so that data loaders can get the values loaded by other data loaders previously. For example:
<@prg> data: { doc: xml(data/foo.xml) index: com.example.MyIndexBuilderDataLoader(get(doc)) } @prg>The <@c>get@c> data loader sees the variables that were already put into the hash in the same enclosing <@s>data@s>/<@s>localData@s> hash TDD expression where it is called, but it doesn't see values that are put into the data model elsewhere, as in other (inherited or inheriting) configuration files. However, if you use <@c>get@c> in the <@s>localData@s> setting, it will also see the session level data (see <@s>data@s>).
@dataLoader> <@dataLoader name="antProperty"> <@param name="propertyName" type="string">The name of the Ant property.@param> <@param name="defaultValue" type="any" optional=true>The value returned if the Ant property does not exist.@param>This data loader returns the value of an Ant property. If no Ant property with the given name exists, it will return nothing (i.e. will not add new variable to the shared data model), or it will return the value of the second parameter if you use that.
The values of Ant properties are strings. But sometimes you want to see a property as numerical variable, or as boolean variable, etc. If the property name you give as data loader parameter ends with <@c>?n@c>, then the string will be converted to number variable, and the <@c>?n@c> itself will not count as the part of the actual property name. The complete list of postfixes:
To see a concrete example, look at <@example 'ant2' />.
This data loader will work only if you execute FMPP as Ant task.
@dataLoader> <@dataLoader name="antProperites"> <@param name="propertyName1" type="string" optional=true>The name of an Ant property to expose.@param> <@param name="propertyName2" type="string" optional=true>The name of another Ant property to expose.@param> <@param name="propertyNameN" type="string" optional=true>...etc. You can specify any number of Ant parameters.@param>Returns the set of all Ant properties, or the set of selected ant properties, as a hash.
If you use this data loader without parameters, then it returns the hash of <@e>all@e> Ant properties. To see a concrete example, look at <@example 'ant' />.
You can also give the name of properties to expose as parameters. For example if you want to put only the Ant properties <@c>foo@c> and <@c>bar@c> into the shared data model, then you write: <@c>antProperties(foo, bar)@c>. Parameters that refer to non-existing Ant properties will be silently ignored. The same postfixes (as <@c>?n@c>) are supported as with <@a href="#key_antProperty"><@c>antProperty@c>@a>.
To see a concrete example, look at <@example 'ant2' />.
This data loader will work only if you execute FMPP as Ant task.
@dataLoader> <@dataLoader name="antProject">Returns the current Ant project object (<@c>org.apache.tools.ant.Project@c>). See the example in <@example 'ant' />.
This data loader will work only if you execute FMPP as Ant task.
@dataLoader> <@dataLoader name="antTask">Returns the current Ant task object (<@c>org.apache.tools.ant.Task@c>). See the example in <@example 'ant' />.
This data loader will work only if you execute FMPP as Ant task.
@dataLoader> @sect> <@sect title="Custom data loaders">If you want to write custom data loader, you have to write a Java class that implements the <@c>fmpp.tdd.DataLoader@c> interface (see in the <@a href="api/index.html">API documentation@a>). Then, if the class is available in the class path, or if you drop its jar into the <@c>lib@> directory of the FMPP installation, you can call it similarly as a predefined data loader, just you use the full-qualified class name as the data loader name. For example:
<@c>com.example.fmpp.SQLDataLoader("SELECT * FROM products")@>.
<@e>If you have written a data loader that can be useful for other FMPP users, I will be happy to make it available on the FMPP home page for download. So if you want to contribute with your data loader, drop me a mail (<@myEmail />) or write to the mailing list. Thank you!@e>
@sect> @page>