Ever since the advent of FileMaker Go for iOS, the FileMaker world has been awash with discussions about sync. We’ve all quickly discovered that although FileMaker Go is great at networking back to a FileMaker Server, and using only hosted files, in reality networks are not that great and the costs of reconnecting or losing data because of dropouts isn’t worth the hassle. Plus local data means you get local speed that cannot be replicated over network connections.
So the answer appears to be sync, but sync is hard. Try searching for “icloud core data sync issues” for a good idea of the sorts of problems that Apple is having with their own sync engine. I’m not surprised that FileMaker hasn’t built a sync engine in the product.
So when FileMaker Go became available we started to look at some of the sync models that are available and what options there are. Very quickly a few FileMaker sync products became available so now there are some good choices in terms of a sync method that suits what most developers would want to do. But we’re not most developers 🙂
Is there another angle?
We’ve long been a fan of using Web Services for interacting with other applications, and with RESTfm for FileMaker Server, it’s a great model for interacting with a FileMaker database as well.
Recently we were talking to Kevin Frank of FileMaker Hacks fame about a sync project of his own. Off handedly, I suggested that a web based sync model would remove the overhead of connecting via FMP networking protocols and could potentially be faster.
And so we dug out some rough code we had for doing sync from Go via RESTfm and started to see if it could be built into a full framework for sync in any solution.
What we’ve learnt along the way.
In version 12, FileMaker added the Insert From URL script step. This seemed like a godsend as the previous idea of using a webviewer and scraping the results was painful. But it’s riddled with issues.
Insert From URL should be a function call not a script step
Firstly it’s a Insert step, going into a field on the local layout. It would be great to have a different way of calling this, I can’t see any reason why it couldn’t be a Set Variable step with a HTTP_GET function, but it’s what we have, so we’ll live with it until plugins work on Go.
Insert From URL needs to support more than GET
Secondly, it’s GET only, there’s no support for POST, PUT or DELETE. We’re lucky with RESTfm that we can override the method and do POST and PUT within the GET, but this probably isn’t an option for any other Web Services where you can’t control the back end.
Insert From URL shouldn’t mess with your URL
Thirdly FileMaker seems to have awful trouble with URLs and seems to insist on altering them before sending to the server. Even if you cleverly encode some values in the URL that need to be encoded like “&”, FileMaker will unencode them for you and probably break your url in doing so. We’ve had to add extra encoding on top of the normal GetAsURLEncoded to allow for this.
We need better error reporting
Fourth, FileMaker reports back some strange errors on this script step. If it doesn’t like your URL FileMaker will fail to send it and return a 1603 error – “URL Format is incorrect”. But even when it likes your URL, if the web server returns a perfectly valid 204 result ( Accepted, no Content ) FileMaker will give you a 507 error, which means “Value in field failed calculation test of validation entry option”. Considering the field has no validation, it’s an odd error. And when the web server fails and gives you a 404 error, FileMaker happily accepts it with no error. You’d need to parse the result and hope there is one to find that out.
Long term if using Web calls is going to be a core part of FMP we need to have a better handle on the errors that the Web Server can respond with, separately from the result of the call itself.
GET is a limited option
But the biggest issue is a limitation in GET itself, in that you are limited to a certain number of characters in your URL. This number is dependent on what Web Server you’re accessing, what version it is and what client you’re accessing it with. For example access IIS 7 with a URL of more than 2048 characters, and it returns a 404 error regardless of whether the url exists or not. On top of this it seems like FileMaker itself imposes a fixed limit on the URL length of somewhere around 2000 characters. So what if you want to do a GET with a 2500 character string in Insert From URL? Well you’re out of luck.
Was this the end of our great idea?
Short answer : No. None of these issues were insurmountable, including the GET limitation. We have the override method in RESTfm, so we can do POST using a GET. We can double encode our URL using only the values that FileMaker breaks so we get back the exact same result we sent out. We have been through every possible Web Server result code and documented how FileMaker behaves. And we can use a Append function in RESTfm to string together multiple requests and send any length string to FileMaker Server.
Part 2 about how it works coming soon…