Goya Blog
Plugin Updates - JSON, dialogs, PUT and Sort
We've been busy working on our free BaseElements plugin again. This is a minor alpha release, but contains quite a few changes.
JSON
First we've added more JSON support in the ability to encode data in the required JSON format. So a new function :
BE_JSON_Encode ( key {; value ; type } )
Allows you to build properly encoded JSON data. Using this one function as a base, you can encode pretty much any FileMaker data into proper JSON. Some structures ( arrays etc ) need a little manual extra work, but the work of having proper JSON data in FileMaker is now simple with both incoming data ( BE_JSONPath ) and now outgoing available.
Value Lists
We've also added some basic list manipulation that really should be in the FileMaker function list, but isn't.
BE_Values_Unique ( listOfValues )
Will trim a list down into only unique values.
BE_Values_Sort ( listOfValues )
Sorts a list alphanumerically.
BE_Values_FilterOut ( textToFilter ; filterValues )
Does the opposite of FilterValues where it leaves in only the filtered values, this one leaves them out. In other words it's like a NOT function for Value Lists - remove anything in the textToFilter that is in filterValues, leaving only the values NOT in the list.
HTTP PUT
Our first iteration of this function only allows you to upload a file. So we've split the functions into two :
BE_HTTP_PUT_DATA ( url ; data {; username ; password } )
and
BE_HTTP_PUT_FILE ( url ; path {; username ; password } )
That will make it simpler to update records in Web Services using the plugin.
Dialogs
We've also tweaked some of the progress dialog functions so they display better, especially on Windows.
Download
You can download the Mac, Windows 32 bit and Windows 64 bit versions today, it is version 2.1.0a4.
Curl Options with our FileMaker Plugin
The new release of our FileMaker plugin includes an option to set some of the various curl options, used whenever the plugin performs a HTTP function call. Although there is a complete list of options that curl provides, not all options are supported by the plugin.
( As an aside, if you're wondering what curl is and why any of this is interesting ; Curl is a code library that provides lots of networking protocol code that other applications often re-use. We use it as the basis for the GET and HTTP functions in our plugin and in fact FileMaker itself also uses it. Open the "FMP Acknowledgements.pdf" file that comes with the latest version of FMP and search for "libcurl". Having this library means each application doesn't have to write its own networking stack from scratch.
The options we're referring to are a whole set of extra flags you can send to curl to change its behaviour. From simple things like a login name, or port number to much more technical details about timeouts and sockets. These things can be very useful in altering the way that http functions are performed. )
Curl options are set with :
BE_Curl_Set_Option ( option ; value )
The function itself doesn't return any value, but you can use BE_GetLastError to get any results. Two notable possible results are 11 for options it doesn't know about, and 13000 for options it does know about but can't handle.
So the complete list of options that we are attempting to handle are :
CURLOPT_PROXY
CURLOPT_NOPROXY
CURLOPT_SOCKS5_GSSAPI_SERVICE
CURLOPT_INTERFACE
CURLOPT_NETRC_FILE
CURLOPT_USERPWD
CURLOPT_PROXYUSERPWD
CURLOPT_USERNAME
CURLOPT_PASSWORD
CURLOPT_PROXYUSERNAME
CURLOPT_PROXYPASSWORD
CURLOPT_TLSAUTH_USERNAME
CURLOPT_TLSAUTH_PASSWORD
CURLOPT_ACCEPT_ENCODING
CURLOPT_COPYPOSTFIELDS
CURLOPT_REFERER
CURLOPT_USERAGENT
CURLOPT_COOKIE
CURLOPT_COOKIEFILE
CURLOPT_COOKIEJAR
CURLOPT_COOKIELIST
CURLOPT_HTTPGET
CURLOPT_MAIL_FROM
CURLOPT_MAIL_AUTH
CURLOPT_FTPPORT
CURLOPT_FTP_ALTERNATIVE_TO_USER
CURLOPT_FTP_ACCOUNT
CURLOPT_RTSP_SESSION_ID
CURLOPT_RTSP_STREAM_URI
CURLOPT_RTSP_TRANSPORT
CURLOPT_RANGE
CURLOPT_CUSTOMREQUEST
CURLOPT_DNS_SERVERS
CURLOPT_SSLCERT
CURLOPT_SSLCERTTYPE
CURLOPT_SSLKEY
CURLOPT_SSLKEYTYPE
CURLOPT_KEYPASSWD
CURLOPT_SSLENGINE
CURLOPT_CAINFO
CURLOPT_ISSUERCERT
CURLOPT_CAPATH
CURLOPT_CRLFILE
CURLOPT_RANDOM_FILE
CURLOPT_EGDSOCKET
CURLOPT_SSL_CIPHER_LIST
CURLOPT_KRBLEVEL
CURLOPT_SSH_HOST_PUBLIC_KEY_MD5
CURLOPT_SSH_PUBLIC_KEYFILE
CURLOPT_SSH_PRIVATE_KEYFILE
CURLOPT_SSH_KNOWNHOSTS
CURLOPT_VERBOSE
CURLOPT_HEADER
CURLOPT_NOSIGNAL
CURLOPT_WILDCARDMATCH
CURLOPT_FAILONERROR
CURLOPT_PROXYPORT
CURLOPT_PROXYTYPE
CURLOPT_HTTPPROXYTUNNEL
CURLOPT_SOCKS5_GSSAPI_NEC
CURLOPT_LOCALPORT
CURLOPT_LOCALPORTRANGE
CURLOPT_DNS_CACHE_TIMEOUT
CURLOPT_DNS_USE_GLOBAL_CACHE
CURLOPT_BUFFERSIZE
CURLOPT_PORT
CURLOPT_TCP_NODELAY
CURLOPT_ADDRESS_SCOPE
CURLOPT_TCP_KEEPALIVE
CURLOPT_TCP_KEEPIDLE
CURLOPT_TCP_KEEPINTVL
CURLOPT_NETRC
CURLOPT_HTTPAUTH
CURLOPT_TLSAUTH_TYPE
CURLOPT_PROXYAUTH
CURLOPT_AUTOREFERER
CURLOPT_TRANSFER_ENCODING
CURLOPT_FOLLOWLOCATION
CURLOPT_UNRESTRICTED_AUTH
CURLOPT_MAXREDIRS
CURLOPT_PUT
CURLOPT_POST
CURLOPT_POSTFIELDSIZE
CURLOPT_COOKIESESSION
CURLOPT_HTTP_VERSION
CURLOPT_IGNORE_CONTENT_LENGTH
CURLOPT_HTTP_CONTENT_DECODING
CURLOPT_HTTP_TRANSFER_DECODING
CURLOPT_TFTP_BLKSIZE
CURLOPT_FTPPORT
CURLOPT_DIRLISTONLY
CURLOPT_APPEND
CURLOPT_FTP_USE_EPRT
CURLOPT_FTP_USE_EPSV
CURLOPT_FTP_USE_PRET
CURLOPT_FTP_CREATE_MISSING_DIRS
CURLOPT_FTP_RESPONSE_TIMEOUT
CURLOPT_FTP_SKIP_PASV_IP
CURLOPT_FTPSSLAUTH
CURLOPT_FTP_SSL_CCC
CURLOPT_FTP_FILEMETHOD
CURLOPT_RTSP_REQUEST
CURLOPT_RTSP_CLIENT_CSEQ
CURLOPT_RTSP_SERVER_CSEQ
CURLOPT_TRANSFERTEXT
CURLOPT_PROXY_TRANSFER_MODE
CURLOPT_CRLF
CURLOPT_RESUME_FROM
CURLOPT_FILETIME
CURLOPT_NOBODY
CURLOPT_INFILESIZE
CURLOPT_UPLOAD
CURLOPT_MAXFILESIZE
CURLOPT_TIMECONDITION
CURLOPT_TIMEVALUE
CURLOPT_TIMEOUT
CURLOPT_TIMEOUT_MS
CURLOPT_LOW_SPEED_TIME
CURLOPT_MAXCONNECTS
CURLOPT_FRESH_CONNECT
CURLOPT_FORBID_REUSE
CURLOPT_CONNECTTIMEOUT
CURLOPT_CONNECTTIMEOUT_MS
CURLOPT_IPRESOLVE
CURLOPT_CONNECT_ONLY
CURLOPT_USE_SSL
CURLOPT_ACCEPTTIMEOUT_MS
CURLOPT_SSLENGINE_DEFAULT
CURLOPT_SSLVERSION
CURLOPT_SSL_VERIFYPEER
CURLOPT_SSL_VERIFYHOST
CURLOPT_CERTINFO
CURLOPT_SSL_SESSIONID_CACHE
CURLOPT_GSSAPI_DELEGATION
CURLOPT_SSH_AUTH_TYPES
CURLOPT_NEW_FILE_PERMS
CURLOPT_NEW_DIRECTORY_PERMS
CURLOPT_POSTFIELDSIZE_LARGE
CURLOPT_RESUME_FROM_LARGE
CURLOPT_INFILESIZE_LARGE
CURLOPT_MAXFILESIZE_LARGE
CURLOPT_MAX_SEND_SPEED_LARGE
CURLOPT_MAX_RECV_SPEED_LARGE
Progress Dialogs using the free BaseElements plugin
We released a beta version of our free BaseElements plugin earlier and I need to document the new progress dialog functionality as it isn't readily understandable from just the function calls themselves.
Barber Pole Progress Dialogs
Firstly there is the main BE_ProgressDialog function :
BE_ProgressDialog ( title ; description {; maximum } )
the title, and description are obvious, and similar to other dialogs that the plugin produces. The "maximum" value is optional, but does all the work. If you leave it out you get an "indeterminate" dialog, so the barber pole effect :
You cancel a barber pole dialog by calling the BE_ProgressDialog_Update function with any value less than zero.
BE_ProgressDialog_Update ( -1 )
Incremental Progress Dialogs
If you have a value for maximum, then you get the normal start and finish progress dialog. There is an assumed minimum of 0, so if you want to count from 10 to 20, then you actually need to count from zero to 10.
To update the progress indicator, call the function with increasing values each time, until you reach the maximum value. Any value above the maximum will cause the dialog to close. The button isn't modifiable, but should have the correct language for your operating system. You can have the button disabled by setting the Allow Abort State to Off.
This is the first iteration of this function, but this is the functionality we were after for including in our own products, but if you'd like it to behave differently or would like other features, please let us know and consider sponsoring a function.
A new FileMaker sync framework using Web Services
Ever since the advent of FileMaker Go for iOS, the FileMaker world has been awash with discussions about sync. We've all quickly discovered that although FileMaker Go is great at networking back to a FileMaker Server, and using only hosted files, in reality networks are not that great and the costs of reconnecting or losing data because of dropouts isn't worth the hassle. Plus local data means you get local speed that cannot be replicated over network connections.
So the answer appears to be sync, but sync is hard. Try searching for "icloud core data sync issues" for a good idea of the sorts of problems that Apple is having with their own sync engine. I'm not surprised that FileMaker hasn't built a sync engine in the product.
So when FileMaker Go became available we started to look at some of the sync models that are available and what options there are. Very quickly a few FileMaker sync products became available so now there are some good choices in terms of a sync method that suits what most developers would want to do. But we're not most developers :)
Is there another angle?
We've long been a fan of using Web Services for interacting with other applications, and with RESTfm for FileMaker Server, it's a great model for interacting with a FileMaker database as well.
Recently we were talking to Kevin Frank of FileMaker Hacks fame about a sync project of his own. Off handedly, I suggested that a web based sync model would remove the overhead of connecting via FMP networking protocols and could potentially be faster.
And so we dug out some rough code we had for doing sync from Go via RESTfm and started to see if it could be built into a full framework for sync in any solution.
What we've learnt along the way.
In version 12, FileMaker added the Insert From URL script step. This seemed like a godsend as the previous idea of using a webviewer and scraping the results was painful. But it's riddled with issues.
Insert From URL should be a function call not a script step
Firstly it's a Insert step, going into a field on the local layout. It would be great to have a different way of calling this, I can't see any reason why it couldn't be a Set Variable step with a HTTP_GET function, but it's what we have, so we'll live with it until plugins work on Go.
Insert From URL needs to support more than GET
Secondly, it's GET only, there's no support for POST, PUT or DELETE. We're lucky with RESTfm that we can override the method and do POST and PUT within the GET, but this probably isn't an option for any other Web Services where you can't control the back end.
Insert From URL shouldn't mess with your URL
Thirdly FileMaker seems to have awful trouble with URLs and seems to insist on altering them before sending to the server. Even if you cleverly encode some values in the URL that need to be encoded like "&", FileMaker will unencode them for you and probably break your url in doing so. We've had to add extra encoding on top of the normal GetAsURLEncoded to allow for this.
We need better error reporting
Fourth, FileMaker reports back some strange errors on this script step. If it doesn't like your URL FileMaker will fail to send it and return a 1603 error - "URL Format is incorrect". But even when it likes your URL, if the web server returns a perfectly valid 204 result ( Accepted, no Content ) FileMaker will give you a 507 error, which means "Value in field failed calculation test of validation entry option". Considering the field has no validation, it's an odd error. And when the web server fails and gives you a 404 error, FileMaker happily accepts it with no error. You'd need to parse the result and hope there is one to find that out.
Long term if using Web calls is going to be a core part of FMP we need to have a better handle on the errors that the Web Server can respond with, separately from the result of the call itself.
GET is a limited option
But the biggest issue is a limitation in GET itself, in that you are limited to a certain number of characters in your URL. This number is dependent on what Web Server you're accessing, what version it is and what client you're accessing it with. For example access IIS 7 with a URL of more than 2048 characters, and it returns a 404 error regardless of whether the url exists or not. On top of this it seems like FileMaker itself imposes a fixed limit on the URL length of somewhere around 2000 characters. So what if you want to do a GET with a 2500 character string in Insert From URL? Well you're out of luck.
Was this the end of our great idea?
Short answer : No. None of these issues were insurmountable, including the GET limitation. We have the override method in RESTfm, so we can do POST using a GET. We can double encode our URL using only the values that FileMaker breaks so we get back the exact same result we sent out. We have been through every possible Web Server result code and documented how FileMaker behaves. And we can use a Append function in RESTfm to string together multiple requests and send any length string to FileMaker Server.
Part 2 about how it works coming soon...
BaseElements 4.0.5
Sometimes BaseElements updates come thick and fast. Other times there's a wait between versions. This is one of the latter.
BaseElements 4.0.2 was released back in July of last year. We've since been working on some minor tweaks and fixes to BE that didn't at any point seem, by themselves, to warrant an new release. But we had also been working on a change to the free BaseElements plugin that allows us to remove nodes from XML documents. This was working with the 2.0.1 update to the plugin back in February.
What's the connection to BaseElements? Glad you asked. The biggest issue we have with the import process in BE is that on very large data sets ( many GB of XML DDR files ) we often hit limits in FileMaker. With the plugin, we don't have access to more than 2GB of memory. Above this we get errors ( which we can trap for ). So we switch our XSLT processing to the internal Xalan based import engine. This works great, except in some cases where not only does it not work, it just stops half way. No errors, nothing is reported, but people contact our support page to say that the import only half worked.
There is no way for us to alter the internal FMP import, as we don't control that part of the process. Until the plugin architecture moves to 64 bit in the client, there's no way to increase the amount of memory available. And until we move to a XQuery based import model we need to load all of the XML file into memory every single time we import, and there are multiple imports for a single Analysis.
So the new 4.0.5 release of BaseElements includes and option to reduce the size of the XML files by removing some nodes it doesn't need, and only importing the end results. We remove the HexData and PlatformData ( print config ) nodes, as well as ObjectStyle, CharacterStyleVector and ParagraphStyleVector nodes. None of this contains anything useful to BE, and the resulting XML files can be up to 90% smaller than the originals.
In all our testing so far, this has been able to fix issues with a DDR that just failed to import before in BaseElements. And in some cases it's also reduced the import times as well. The original files aren't touched, and the import result would be exactly the same either way.
This is a new checkbox on the setup layout inside BE :
It's off by default, so if you're having this issue, or would like to see if it reduces import times on your large solution, give it a try. And if you find anything useful or have any issues to report, please contact us.
There's a whole slew of minor fixes in this version of BE, so we suggest everyone using v4 should update.
More Plugin Features
We continue to add to our free FileMaker plugin. This time we've been doing lots of little bits and we've got a couple of great new features we need to test before going final with our next update.
Some simple list functions
We've added :
BE_Values_FilterOut ( textToFilter ; filterValues )
BE_Values_Sort ( listOfValues )
BE_Values_Unique ( listOfValues )
FilterOut removes values from a list, so you can then get the ones that remain. And Sort and Unique should be fairly obvious.
OAuth
First of all is OAuth support. OAuth is complex, and we've tried to do this in a way that you can utilise the existing HTTP functions with the OAuth based authentication. To use this, there is a new function :
BE_OAuth_RequestAccessToken ( url ; consumer_key ; consumer_secret ; request_key ; request_secret )
Call it once with empty "request" values, to get the request key and secret. Then call this function again with the new values to get the access key and secret. Hopefully then the BE_GetURL functions will use the new authentication and not require a username and password. Call that function again with empty values to clear it all out.
There's probably some work to do on this get it all sorted and some error reporting to be done, but if you've been wanting to access an OAuth based Web Service, this will be a godsend.
Curl Options
Also because all of our HTTP functionality that we've implemented uses the free curl library at the back end, we've been able to add a quick function to set all of the curl options :
BE_Curl_Set_Option ( option ; value )
Call it with an option name, and a value, or empty values to clear them out. This will be handy for some of those edge cases where you need specific details for accessing the server.
Progress Dialogs
And finally we've added progress dialogs for allowing a simple way to show progress in a process. This is in a few of the other full featured dialog plugins, but is something we've wanted for a while.
If you're testing the new features and have any feedback or issues, please contact us with your results. And if you try them out and they work and you like them, we'd love to know that too, it's helpful to know they're working just as much as it is to know they don't work.
Download
You can download the Mac, Windows 32 bit and Windows 64 bit versions today, it is version 2.1.0a3.
Integrating FileMaker and Drupal via Web Services
There are a few ways of having drupal talk to FileMaker Server that we've used in the past before. Our demo registration forms interact with FileMaker via the WebForm and FileMaker form modules. This works great for this one particular situation of filling in a form and having that data sent to FileMaker Server.
But for more complex interactions, the best approach is going to be a mix of Web Service operations. There is a great post and webinar up on the MediaCurrent website about how they integrated Drupal with FileMaker using our very own RESTfm.
This is exactly the sorts of things we originally had planned for RESTfm, so it's good to see it getting some traction out in the wider world.
RefreshFM session at DevCon this year
Goya has done Vendor Sessions at the last two previous FileMaker Developer Conferences. Both sessions were primarily about BaseElements and how to get more from that product.
This year we decided to take a slightly different approach and we're going to talk about solution updates using RefreshFM. You can see details of the session in the schedule, it's at 2pm on the first day.
So if you're interested in learning more about RefreshFM or about solution updating in general, then we'd love to see you there.
Sending data to RESTfm from PHP
We use RESTfm a lot to send quick data from FileMaker to a FileMaker Server box without going through all of the time delays of having external FileMaker files open for scripts or relationships. Having the ability to do a quick GET or POST into FileMaker saves a lot of time.
We think RESTfm could be a great candidate for speeding up sync from FileMaker Pro or Go to FMS, but that's a post for another time :)
In the mean time I needed some code in PHP for a different use, that sent data to FileMaker via RESTfm. So I've written this up as sample code. If you're needing to send records to FileMaker from php, this this is hands down the simplest way to accomplish it.
Attached is a copy of the sample code, pushing records to our postcodes database. Feel free to have a play and adapt it to your own use.
| Attachment | Size |
|---|---|
| post.php_.zip | 2.51 KB |
FileMaker and JSON
FileMaker has long has support for XML in it's import and web processes and quite often people are building Custom Functions to parse XML within FileMaker. We now use our free BaseElements plugin to do XPath inside FileMaker, and it's really handy to get at details of XML without having to go through an import process.
But JSON is fast becoming as popular as XML for all sorts of data these days. Quite often it's available in Web Services as an option along side XML and I'm now starting to see it appear as the only option in a few places.
In order to take advantage of this we've decided to move to add some JSON functionality to our BaseElements plugin. We have a new build, coming quick on the heels of the 2.0 release, we've added :
BE_JSONPath ( json, query )
BE_JSON_ArraySize ( json )
BE_JSON_Error
We think this will be useful to FileMaker developers who work with Web Services or just want a way to encode data for other processes.
You can download this now for Mac, Windows 32bit and Windows 64bit for FMS CWP.
