I've been working on an update to RefreshFM for a while now and one my my todo items has been to sort out a way to include external storage in the import process. Imagine my surprise a few weeks ago, when my testing showed that not only did my idea for an import method work, but that you didn't end up with duplicate files.
When you have an good idea, you are always hopeful it will work, but I genuinely thought it would duplicate the external files. I wasn't looking for any other results, so when it didn't duplicate the files, I was very much surprised.
So I duly sent out some updates via the newsletter, in time for people to sign up for the FM Product Conference. In the mean time I kept working on the rough edges on the new changes to RefreshFM for 3.0.
And finally with a new beta ready to release to the public, I went back over my External Storage setup, and to my horror, it was now duplicating files. After much fiddling, testing, adjusting, tweaking and frustration, I can only conclude that the initial result I saw of non-duplicated files was a mirage. A tantalising, exciting mirage, but a mirage nonetheless.
So, to everyone who's hopes I mistakenly raised about imports without duplication, my sincere apologies.
No matter how much I try, I can't repeat the setup I had. In all of my testing since, I am unable to import without duplicates. I won't leave this unresolved and will continue to try to find a method that works and that doesn't require duplication, but I suspect that this may be something out of the control of the developers, and something that requires an internal FMP change.
To everyone who is still keen to look at an import method that includes External Storage, but requires duplication of the external files, we have a download now available of the public beta of RefreshFM 3.0b3 for Mac only. The documentation is the next thing for me to work on, but if you're familiar with 2.0, then 3.0 will be a fairly simple update.
There are just a couple of things left to do before the final :
- Complete the Windows Automation.
- Finalise the new user UI.
We have a plan in place and code started on the Windows automation, so it will be very exciting to have this aspect of the process cross platform. And we have some great ideas about a new UI for end users for the distribution version, which I'm excited about putting into place.
Most of all, if you've ever tried RefreshFM before and not quite understood what it needs to "setup" then this will be the release for you. Our automated setup process ( Mac only until the next beta ) is awesome. If I do say so myself.
RESTfmSync makes heavy use of the new in v12 "Insert From URL" script step to send data to the FileMaker Server. This works very well, but has a few limitations. Seeing as we've been through all of the testing and trials to work out exactly what those limitations are, we thought it worth documenting them for others using this step in their solutions.
GET vs POST
First of all, the Insert From URL ( hereafter referred to as IFU ) step only supports a GET operation. It can't do PUT, POST or DELETE. So some interactions with web servers are not possible. In fact most web APIs utilise much more than just a GET, and so for lots of things Web Service related, IFU just isn't enough.
We have the advantage with RESTfmSync of controlling both the client ( FileMaker Pro on the desktop or FileMaker Go on iOS ) and the server ( RESTfm via Custom Web Publishing in FileMaker Server ). So we've built into RESTfm the ability to override the method. This means we can send a GET request but have RESTfm treat it like any a different request. This is done via an extra parameter
When you send this parameter in your URL you can do a POST via a GET from the Insert From URL step.
GET has length issues
GET also has limitations in terms of the length of the url string that it will accept. Unfortunately this varies with both the web server and web client, so there's no real way of knowing what it will be in advance and you need to limit your data length to the worst case option. In RESTfmSync we've chosen a 1950 character limit as we found that some values of 2000 characters or more would fail in some setups.
Again we can work around this in RESTfm by appending data and sending multiple requests to the server. Using the
parameter, you can elect to have the data sent be appended onto the end of the fields instead of replacing them.
Insert From URL likes to mess with your data
However the most frustrating limitations in IFU are that it modifies some data in an effort to help you, but that only really hinders things. It has some encoding built into the step that modifies the data in your url before it's sent. The idea seems to be that it auto encodes things for you, meaning you don't have to encode it yourself. However it also means it encodes the wrong things.
When the WebViewer was introduced, it also had this same behaviour, and in FileMaker 11 and later, there was an optional "Automatically Encode URL" checkbox introduced that defaults to on.
The help describes this as :
Select Automatically encode URL to allow FileMaker Pro to apply encoding rules to the URL, if necessary, so that it complies with a browser’s required format. To keep the URL in the format in which it is entered, deselect this checkbox.
And it goes on to further explain :
- The following characters are never automatically encoded: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz 0123456789-_.~!*'();:@=+$,/?
- An ampersand (&) is encoded only if a space follows it. For example, “& “is encoded, but “&x” is not.
- A pound sign (#) is encoded only if a number character (0 through 9) follows it.
- The backslash (\) and percent (%) characters are always encoded.
- All other characters are always encoded.
However, none of this mentions the IFU script step, just the WebViewer. But it's clear there is some encoding being done to the your data in IFU as well.
For example, if you want to send to a url a new field value of "abc&def" for the field called "MyData" you would send the following :
But you can't send the & as part of the data, the & separates field values, so it needs to be encoded. The encoded value for & is %26. FileMaker Pro will even do this for you via GetAsURLEncoded ( "abc&def" ). So you actually need to send :
But if you're paying attention to the encoding rules above, FileMaker decides that the % needs to be encoded as it's URLEncoded value and so even though you're sending %26 it converts this silently behind the scenes and sends:
And your MyData field ends up with the value of "abc%26def".
In most situations this is completely unhelpful as it makes it impossible to distinguish between text that is an encoded ampersand and text that is actually %26 and is meant to show that. And there's no way to turn this off.
RESTfm to the rescue again
We're fortunate in RESTfm to be able to work around this because of our ability to control client and server. Inside RESTfmSync, we use a Custom Function that does the following:
GetAsURLEncoded ( Substitute ( text ;
[ "%" ; "%25" ] ;
[ "!" ; "%21" ] ;
[ "#" ; "%23" ] ;
[ "$" ; "%24" ] ;
[ "&" ; "%26" ] ;
[ "'" ; "%27" ] ;
[ "(" ; "%28" ] ;
[ ")" ; "%29" ] ;
[ "*" ; "%2A" ] ;
[ "+" ; "%2B" ] ;
[ "," ; "%2C" ] ;
[ "/" ; "%2F" ] ;
[ ":" ; "%3A" ] ;
[ ";" ; "%3B" ] ;
[ "=" ; "%3D" ] ;
[ "?" ; "%3F" ] ;
[ "@" ; "%40" ] ;
[ "[" ; "%5B" ] ;
[ "]" ; "%5D" ] ) )
This way at the other end we can undo that extra encoding, and then url un-encode and get back the original data. We detect this whenever the parameter RFMfixFM01 is included. If it's not included the it is assumed that it's a normal encoding.
The obvious issue with this is in situations outside of a managed client/server environment, such as external public APIs. You have no control over the server and so some data just cannot be sent accurately.
For RESTfmSync, we can work around the issue, but it means that one character ( & ) is turned into five ( %2526 ) and so makes it much more likely that we'll run into the URL length limitations.
The advantage of the IFU step is that it can happen in the background on a layout where the field being referenced is off layout and not seen by the user. Our alternative option in RESTfmSync is to use the WebViewer to do the POST instead. This works, and has none of the limitations of the length of the url, or strange encoding issues. It does however force us to have the WebViewer visible to the user, and also requires us to have a scripted callback mechanism that the Wv calls to complete the process.
It would be really nice to see FileMaker build in native functionality for the sorts of HTTP calls that we can do in the BaseElements plugin.
Our hope with the plugin is that it is made obsolete by advances in FileMaker itself. FileMaker has already obsoleted two of our functions, which were the SQL calls to grab Table and Field structure directly from the file. Being able to act as a full Web Services client would be a great next step on that path.
In the short term a "Automatically Encode URL" checkbox for the Insert From URL step would be a great help.
A question that has come up more than once about the progress dialogs in our free FileMaker plugin is how to work with the cancel buttons.
Showing or Hiding the button
The cancel button is optional, and appears based on the status of the "Allow User Abort" state. When this is Off, the button won't appear and when Abort is allowed, then the button appears. So you can control the visibility of this within your script by altering the state.
Trapping for Cancel button clicks
When the cancel button is clicked, it closes the dialog, but you then need to capture that state. Presumably you're doing some looping script, or a long process with lots of steps. Either each time it loops or at some point in the loop, you're calling the BE_ProgressDialog_Update function to update the progress.
When the Update function call is successful, the BE_GetLastError function will return 0. When the user has clicked the cancel button, the BE_GetLastError function will return 1 which is equivalent to the FileMaker "User canceled action" error code. Trapping for this in your code allows you to gracefully exit your loop and react to the cancel.
RefreshFM 2.0 is finally released and the last update since the beta added a function that has always been a request of mine in Import dialogs within FileMaker Pro itself : the ability to choose "Matching IDs" in the field mapping.
Using this option in RefreshFM, you can create an import order that still works if you delete fields, and add fields, and rename fields from either source or destination table.
This works in RefreshFM because we build the import script ourselves, and with FileMaker Pro 12 there's a change to the way copy and paste works in that it now respects the order you copy, instead of just switching to "Last Order" like FMP 11 and earlier did.
So we can dynamically build any import order we like. Having a choice of matching names is good, I think having matching ids is even better, and I hope this concept comes into the FileMaker product line at some point. Although an overhaul of the import dialog would be the best outcome, as there's quite a few other suggestions I'd make for that.
RefreshFM version 2 is an update for fmp12 format compatibility, along with some other minor additions and fixes. It's a free update to version 1.0, so any licences for 1.0 will continue to work in 2.0.
New licences are $399 for the developer version, and $699 for the distribution licence for those who are selling standalone solutions and need their end users to run the update process.
I'm doing a DevCon vendor session on RefreshFM on Tuesday at 10:45 am if you'd like to see the product in action.
Trying to explain what RESTfm is to someone who hasn't used Web Services before can be difficult. It's hard to get an idea of what RESTfm is, let alone what you'd use it for. ( If you want to be really confused, try reading about REST on wikipedia. )
So we're very excited that our new 2.1.0 version includes some great sample code that you can use in your own solutions that will show you some really cool things you can use RESTfm for.
We're calling this RESTfm Tools. It's going to be an ever growing collection of code samples of things we've developed for our own use, or for customers using RESTfm already, and that will be included in every RESTfm release.
I've talked on the blog before about our sync framework and this is a very exciting release. We've got this sync code working in our own solutions on Pro and FMGo as well as in other developers projects and it's been a great resource. Best of all you can implement this without any other costs after your single RESTfm Server licence.
RESTfmIPN and RESTfmFSValidate
If you're selling a FileMaker solution using PayPal and want to be able to do real time payment processing in your FileMaker database, then the RESTfmIPN solution is for you. It allows you to respond to the IPN request and send the details of a purchase to your FileMaker database via RESTfm.
And alternatively if you use FastSpring for payments instead you can use the RESTfmFSValidate code to do checks of a existing licence detail against your database for validating upgrade pricing.
The new update has now been sent to existing users of RESTfm, but if you didn't get yours, get in touch with us. All of the sample files are in the "code" folder in your RESTfm download.
And if you'd like to see RESTfm or our new sync code in action, come and visit us at our booth at FileMaker DevCon in San Diego next week.
Our RESTfmSync solution has been featured in the latest FileMaker newsletter. The July - September 2012 newsletter includes a story about how Kevin Frank worked to build our FileMaker syncing platform into a solution for one of his clients.
This is a great FileMaker story, about how iPads are changing the ability of companies to deliver results. Especially interesting is how they were able to to sell 28% more boots using this system, and that 98% of people gave their email address for followup communications.
We've been working with Kevin for a few months now, continually refining the RESTfm framework, and extending it and improving it each step of the way.
If you're interested in seeing RESTfmSync in action, contact us for a copy.
We've updated our plugin documentation to include the new JSON, ValueList, Dialog and HTTP function details.
Part of this new documentation are some example uses for some of the more technical functionality. I hope these go a little way towards helping people implement the plugin in their own solutions.
If you'd like to see examples for other functions, or could contribute to the examples show, please let us know, or edit the wiki page directly.
We're excited today to announce that RESTfm has reached version 2.0. Although we haven't previously spent much time talking about RESTfm, we've slowly been working on other tools that interact with it. In the coming days in the leadup to FileMaker DevCon, I'm going to go into more detail about our implementations of :
- PayPal Instant Payment Notifications to RESTfm
- FastSpring software Licence retrieval from RESTfm
- Doing a POST in the WebViewer in FileMaker Go to RESTfm
- Syncing from FileMaker Go to RESTfm via HTTP
Keep an eye on the blog here for more details about these.
The 2.0 release had some big features we needed to help us build our sync framework, notably :
- The RFMelsePOST flag to allow us to do a single request for new or updated records.
- RFMappend for doing large data sets over a small GET connection.
- RFMfixFM01 to work around the way FileMaker handles encoded characters in InsertFromURL.
- A Field meta data flag so you can just retreive the field details without getting any records.
- Our new .simple format for easy to parse data inside FileMaker where there is no JSON or XML.
Plus a bunch of smaller things and bug fixes. All of the details are on the change log.
This is a free update to 1.0, and anyone who has an existing licence will be emailed a copy of the latest release shortly.
Our Sync Framework
We'll be sending out a copy of our sync framework to people who own RESTfm 2.0. If you'd like to help us test and refine the sync before our final public release, please contact us.
RESTfm at first seems like just a thing for WebServices. But when you realise how easy it makes data sending and retrieval, you immediately start finding other uses for it. Mostly what we're seeing is that any small communication with our FMS box almost always defaults to using RESTfm to begin with. We keep finding new uses for it, and it's becoming a very handy tool.
- JSON Encode and Decode
- New Value List manipulations
- HTTP PUT for data
- CURL option values
- Progress Dialogs
One thing that didn't make the beta was some trial code for OAuth. Although the code is still there, the functions won't show in the function list. This just hasn't been widely tested yet and we really wanted to finalise the JSON work for some upcoming sync work we're doing.