Following on from part 1.
The second big change to the XSLT engine in BaseElements 3 is that we consolidated a lot of areas in the system. All of the old field, Custom Function, TO, ValueList, Variable and Plugin reference tables were all joined into a single “References” table. Although having one base table instead of 5 doesn’t make things much faster inside BE, it meant we were able to do a single xslt run instead of five.
For example we’re traversing the entire XML tree to find all the calculations. We do this again to find all of the references within the calculations. But if you imagine that a calculation can reference fields, CFs, TOs, variables and plugins, then repeating that traversal five times for each one is time consuming. So by putting the references into a single table we can do that just once.
But that’s not all, we’ve brought into the xslt code some work that was previously being done as processing steps in BaseElements. If you’ve watched a BaseElements import, you will have seen some steps where it mentions “Processing Calculations”. This was doing a few things : Digesting the calculation steps and producing plugin references and variable items, and also sorting out Indirection steps with function calls. It’s a time consuming step in BaseElements 2, and can be up to a quarter of the total processing time. To give you an idea of the work being done, here is a screenshot of the code from BE2 :
In that there are lots of recursive custom functions ( great for simplifying code, not so great for massive data processing ), and lots of loops that run off and generate new records.
In BaseElements 3 this entire block of code is gone. In it’s place we’ve built in the same functions into the References xslt file. I’m going to digress into some detailed xslt so if you’re not interested, feel free to skim the technical steps or tune out 🙂
The first thing we needed to do was to be able to differentiate plugins from normal functions. FileMaker helpfully puts function calls in a separate part of the DDR, so we can track those, but it doesn’t differentiate between plugins and other functions. To do this in xslt you build a lookup table reference :
<xsl:key name="plugin-lookup" match="PluginList/Plugin/Function" use="@name"/>
then you setup a variable to reference the external file :
<xsl:variable name="plugin-file" select="document('BE_Plugins.xml')" />
And finally when you’ve got a function call to lookup you look it up in your external table :
<xsl:for-each select="$plugin-file">
<xsl:if test="key('plugin-lookup', $FunctionName)">
<xsl:call-template name="RESULTSET"/>
</xsl:if>
</xsl:for-each>
We’ve used these key lookups in a few places other places as well and it makes a huge difference to the processing time.
The second part of that long script was the processing of variables used. FileMaker doesn’t even distinguish variables from other parts of the text, and so they’re just lumped in with text, math symbols and other whitespace as a “NoRef” type. As well as that, variables are now also in the text on layout objects, inside file references, perform find steps and set variable steps. Each one is slightly different, but essentially the process is the same for all of them : sort through the text and find the variables. This is actually harder than it seems. First, you don’t want to find variables inside comments, so you need to remove those. Then you need to remove anything inside quote marks, but you need to first strip out any escaped quotes. Then you need to split the remaining text into dividers. Because a variable name can have spaces, the only way to tell when it ends is to just find the next non-variable name character after the $. This could be any of of the various math symbols or other characters that end a text block. So finally you have a list of the remaining text, and you convert those to nodes and loop through them to find any that start with a $.
This process is more complex to setup inside xsl than it is to code in FileMaker, but the end result is that the process runs much faster because you’re not looping through records and generating multiple new variable reference records. The entire process fits inside the main reference xslt and is included in the single import that imports those.
I won’t post the code here, but if anyone is interested in how it works, let me know and I’ll document it in more detail.
It’s these two changes, as well as the csv imports, that constitute the bulk of the reduced import time in BaseElements 3. It’s funny, I thought I’d done a good job of making things faster in BE 2, and I wasn’t sure where any extra savings would come from. It was only after the v11 update broke our old xsl that I went back to the drawing board and found other optimisations with the new xsl engine. So maybe if v11 still worked with the old code BE 3 might not have been as fast as it is…