Goya Blog

FileMaker integration with Dropbox

This is the first in our series on Web Services integration with FileMaker that I mentioned last week.

DropBox

The API we are going to look at today is DropBox. With this API a FileMaker solution can manage and share files, folders and Dropbox Paper documents.

As an example, when someone signs up to sponsor the BaseElements plugin, we share with a them a DropBox folder with every historical version of the plugin, as well as private beta versions with new functionality.

We do that with a FileMaker script that uses the BaseElements plugin to do HTTP calls to the Dropbox API. If you missed the first article of the series, it is an introduction to the world of web services and FileMaker.

Each interaction with the web service starts with a request from FileMaker sent to a specific url (or endpoint) along with the request parameters.

A request URL will look something like this:

https://api.dropboxapi.com/2/files/list_folder

This endpoint returns the contents of a folder.

The URL breaks down (reading backward) to: i want to use the list_folder function from the files category of the version 2 of the dropbox api. Makes sense? The first part will be the same in all our requests, the only parts that will change will be the function and the category.

Of course we also need to tell the Dropbox API what folder we are talking about and that is done through the parameters (or body) of the request, using JSON:

{
   "path": "/Goya/Secret_Projects"
}

One detail to remember about the Dropbox API is that it uses POST requests even for endpoints where we get data.

So the previous example called from FileMaker with the BaseElements plugin will be

BE_HTTP_POST ( "https://api.dropboxapi.com/2/files/list_folder" ; "{ \"path\": \"/Goya/Secret_Projects\"}" ; "" ; "")

If you go ahead and try that the API will send back an error (at least we know they answer!).

The request can't be processed because we haven't yet told the API what dropbox account we want to use.

So let's go back to our workflow for the Goya sponsors: we have an email address (we receive it from another API) and we want to share our folder with them.

Based on what we said, to successfully process the request Dropbox needs to know:

  • that we are authorized to interact with that Dropbox account
  • what folder do we want to share
  • who are we sharing the folder with and what privileges will they have on the folder and files

Authorization

To identify what Dropbox account we want to work on we need to tell the API that we are authorised to work on it. You do that by getting a token.

The token can be obtained going to this address https://www.dropbox.com/developers/apps and following the steps to register an app and create a token for that app.

Once we have a token, we can add an Authorization HTTP header to the request, and again we can do this with the BaseElements plugin :

BE_HTTP_Set_Custom_Header ( "Authorization" ; "Bearer " & Table::API Token )

Another header we want to add is the language we are using to send the body of our request. In our case it is JSON :

BE_HTTP_Set_Custom_Header ( "Content-Type" ; "application/json" )

Get the Folder list

We need to know the ID of the shared folder so that we can add our new sponsor to it.

The endpoint https://api.dropboxapi.com/2/sharing/list_folders returns a list of all the shared folders from the Dropbox account.

In the body we can specify a limit for the number of items returned.

The following call will return up to 1000 shared folders

BE_HTTP_POST ( "https://api.dropboxapi.com/2/sharing/list_folders" ; "{ \"limit\": 1000 }" )

The API will have one "entry" node for each shared folder

{
    "entries": [
        {
            "access_type": {
                ".tag": "editor"
            },
            "path_lower": "/secret projects",
            "name": "Secret Projects",
            "permissions": [

            ],
            "policy": {
                "member_policy": {
                    ".tag": "anyone"
                },
                "resolved_member_policy": {
                    ".tag": "anyone"
                },
                "acl_update_policy": {
                    ".tag": "editors"
                },
                "shared_link_policy": {
                    ".tag": "anyone"
                },
                "viewer_info_policy": {
                    ".tag": "enabled"
                }
            },
            "preview_url": "https://www.dropbox.com/somelink",
            "shared_folder_id": "123456789"
        },
        ...
    ]
}

We can parse the JSON using the function BE_JSONPath. So to grab the ID of the first shared folder we'll write

BE_JSONPath ( $json ; "$.entries[0].shared_folder_id" )

Share the folder

Now that we know the token, the shared folder id and the email we want to share with, we are ready for the last call to the API.

The endpoint to add a new member to a folder is, surprisingly, https://api.dropboxapi.com/2/sharing/add_folder_member.

The body of the request will look like this

{
    "shared_folder_id": "123456789",
    "members": [
        {
            "member": {
                ".tag": "email",
                "email": "salvatore@goya.com.au"
            },
            "access_level": {
                ".tag": "viewer"
            }
        }
    ],
    "quiet": false
}

where we specify the folder id, the email of the new member and what access privileges they will have.

Calling our BE_HTTP_POST with this url and body will add the member to the folder.

Success! This call doesn't return anything if successful, so we'll have to check the HTTP response header to know if everything went well.

If BE_HTTP_Response_Code returns 200, we know the email was added.

These are only two of the actions available through the Dropbox API and they were all we needed for this workflow.

A complete example file is available to sponsors of the BaseElements plugin, and is sitting in their DropBox share folder already.

Up next in the series, integrating FileMaker and Zendesk.


Salvatore is our web services and integration expert, passionate about linking any API useful to our clients (or us!) to FileMaker. He loves to share ideas and a good story, and as a speaker at DevCon he manages to do both.


FileMaker and web services (and us)

Like most modern companies, here at Goya we use web-based tools. Slack, Dropbox, Zendesk and Mailchimp are awesome platforms to organise our work and to share our ideas.

The best part though is that we can manage all these services from a FileMaker solution.

Which is exactly what happens when someone joins one of Goya's sponsor programs: their Dropbox, Zendesk and Mailchimp account details are auto created via FileMaker scripts.

GoyaSponsor.png

Pretty neat, it saves us quite a few clicks!

We are able to call the web services from FileMaker using the HTTP functions from the BaseElements plugin, and then parse the results using the JSON ones.

Want to try it yourself? In the coming weeks we will publish articles here and add examples to the share folder available to all our sponsors. If you'd like copies of our example files, please consider signing up as a sponsor.

Next week FileMaker integration with Dropbox.

In the meanwhile, if you're not sure this should matter to a FileMaker developer or what web services are, you can download the slides of my Devcon 2015 session
or check out this excellent introduction to web services from Mason Stenquist at DB Services.


Update : Part 1 is up : FileMaker integration with Dropbox


Salvatore is our web services and integration expert, passionate about linking any API useful to our clients (or us!) to FileMaker. He loves to share ideas and a good story, and as a speaker at DevCon he manages to do both.


BaseElements Update and more

It's been a while since the last BaseElements update, and I've had a few emails asking me what's going on. Rest assured we're still working on it, and things are progressing.

And FileMaker 15 is out today, and happens to coincide with me finishing up a big set of changes to BE, so an opportune time to post an update. So to answer all the big questions :

Are you still working on it?

Absolutely. We still use BE every day ourselves, and even use BE to develop BE and couldn't live with out it, much like a lot of you. We haven't sat still but development has been slower while we've worked on client work, RefreshFM updates, RESTfm updates and additions and bug fixes to the Plugin.

Why is importing objects so slow in b12?

What's mostly got us stuck is some changes to the Alerts that we introduced with the new 14 re-design. In order to better track the myriad of issues that can crop up, and be better able to handle errors, warnings, performance issues and unreferenced items, we've been consolidating them into a single alert. So we've added some extra items and included more detail for others, plus combined things into a single location.

And as every developer knows, when you add to a solution iteratively over a long period, all your changes add up. And what we ended up with was a solution for layout objects that was way too complex than it should be to get the job done. At some point in the last few betas our alerts auto enter calculation was doing a sum of a related field, which itself was un-stored, and that will never be fast.

You never set out to design a slow solution, and you never deliberately make a change that is going make things slow, but occasionally you add a great feature that has a cost you didn't plan for.

The solution was to go back and re-think the whole way alerts are generated across all the elements, but especially the large ones ( objects, steps, calculations ). And to do that meant putting a lot of these calculations into the XSLT so that the import was as simple as possible, and therefore as fast as possible.

If you want fast imports, remove all your unnecessary auto enters, lookups, stored calculations and indexes.

We've spent a lot of time re-factoring our XSLT to shift a lot of the complexity there instead of the import. This is a complex, difficult process - XSLT is handled differently in the plugin, vs the command line, vs the import, vs the debugger tools. This makes it really time consuming to debug and test any change.

Our base line for speed testing was importing our sample files ( BaseElements itself ) into the current BE 4. That takes about 5 minutes. We knew that the latest beta of 14 was slower, quite a bit slower. We needed to at least get back to where things were with that version.

At various times during the changes we had imports that took one minute, only to realised hardly any data had come in, and 7 minutes only to make a simple namespace change that cut that in half.

Finally we've got all our data, all working as we want and import times are ... drum roll ... 3 minutes. About a 40% savings on BE 4 and much faster than BE14b12. Excellent, not only did we fix the b12 issue, but we also improved on the previous version.

What's with the beta now that 15 is out?

Our plan with BE 14 was that we'd finish the beta late last year, and then release BE 15 on the release of FMP 15. Obviously that didn't go to plan.

We're fortunate that there are almost no changes to the DDR with 15 so our current release version of BE 4 will work with the FMPA 15 DDR just fine. So will the BE 14 beta.

So we're going to re-start the BE versioning at 15 and go back to b1 and pretend that the whole BE 14 beta never happened :) Well not quite, but it doesn't make sense to have BE 14 around with FMP 15.

There is no upgrade cost from BE 14 to 15. So we will now work on finishing up the remaining features we were working on ( reports mostly ) and release BE 15 as what we were going to do with 14. Everyone who has bought BE back since mid last year will get the final release of BE 15 at no charge and all the current BE 14 licences will work just fine with BE 15.

What else have you done?

One of the biggest support issues we have is in bad XML that FMPA generates that is invalid and so can't be imported. We are pre-processing a lot of that and removing some of these characters and so a lot of imports that would have failed in older versions will work in BE 15.

What else are you going to do?

We've still got some tweaks to how the UI works in BE 15 that we need to work on. There is the reports to do, and we've got some great ideas about those that we think are going to be awesome.

Thanks to everyone for your patience, and we hope it's not going to be much longer till the BE 15 final. In the mean time grab a copy, and send us any feedback.

BaseElements beta 3

We've posted a new version of the BaseElements 14 beta, the new download is b3. Although I was hoping to have a final release done by now, it's taken a bit longer than anticipated.

This version fixes the windows import issue that quite a few people saw, so I've put back the mac and win runtimes. There is a beta version of the plugin out for this.

Great new addition to this version is a new version of fmSearchResults. This is a beta release, with a heap of new functionality, so I'll post more about it soon, but it's good to have this back.

Hopefully this week will finish up some of the outstanding issues - reports, and some of the error notifications, and then we should be right to wind up the beta and start looking towards new functionality.

BaseElements Plugin Updated to 3.1 adds SMTP and more

And one more release note, we've updated the free BaseElements Plugin to 3.1.0. This is a huge release, lots of cool new features.

This version removes support for Windows XP, so be aware of that if you're running older OS versions.

But in return you get SMTP along with JPEG compression for reducing container picture sizes. There's a ConvertContainer for turning a file to a picture and vice versa, so no more exporting and re-importing. There's the ability to evaluate JavaScript inside the plugin, which opens a whole world of possibilities. And BE_ImportFile for importing files into containers - you can do this in the client, but now you can also do it on the server via the plugin.

The documentation will be updated in the next few days, I'll post another note here when it's done.

Sponsors

Thanks again to all the people and companies that sponsor the plugin. It's because of their support that you get all this for free, so if you use the plugin and know someone who sponsors, make sure to buy them a round.

And if you can, please consider sponsoring. It doesn't cover all our costs, but it does help, and you get a proper support channel and a bunch of other cool plugin related code.

New Functions

  • BE_ContainerIsCompressed
  • BE_ConvertContainer
  • BE_Curl_Trace
  • BE_EvaluateJavaScript
  • BE_Gzip
  • BE_ImportFile
  • BE_JPEG_Recompress
  • BE_SMTP_Send
  • BE_SMTP_Server
  • BE_UnGzip
  • BE_Values_ContainsDuplicates
  • BE_Xero_GenerateKeys

Other Changes

  • BE_Curl_Set_Option: label function parameters as optional
  • BE_Curl_Set_Option: the options CURLOPT_HTTPAUTH & CURLOPT_PROXYAUTH take named constants
  • BE_HMAC: add a parameter to allow and input encoding for Base64 and Hex
  • BE_HTTP_POST: allow post args to specify file paths using the @ syntax
  • BE_HTTP_POST: do not crash when a http post contains a parameter that is not a key=value pair
  • BE_JSON_Encode: correctly handle leading 0 for real numbers
  • BE_Values_Unique, BE_Values_FilterOut: add option to be case insensitive
  • Ensure that Get ( InstalledFMPlugins ) returns the same value on both OS X and Windows
  • Remove support for Windows XP

BaseElements 14 beta now available

BaseElements has grown up so quickly it's skipped 9 whole version numbers. Straight from version 4 to version 14, in order to better keep track of compatibility with the current release of FileMaker Pro.

We have a beta version available :

http://www.goya.com.au/files/baseelements/14.0.0b1/BaseElements14_Mac_Runtime.zip
http://www.goya.com.au/files/baseelements/14.0.0b1/BaseElements_14.zip

I started talking about the design of BaseElements in an earlier blog post, and I'll have a lot more to say on that, including a followup about our reasons for making the changes we have.

But for now I wanted to get this beta out so people at DevCon can have a play and see for themselves.

Upgrades

This is a paid upgrade, you can purchase a discounted upgrade during DevCon but the discounts won't last, so get in quick.

RefreshFM 14 beta now available

We've uploaded a test version of RefreshFM ready for FileMaker 14. The only changes in this release are in the automated setup, which now works in FileMaker 14 and 13.

This is a paid update, and you may have noticed that we skipped a few versions, going from 3 to 14. We decided it was better to match the FileMaker Pro version directly so it's clear what release of FileMaker is required.

Download

You can download the new version here :

http://www.goya.com.au/files/rfm/14.0.0b1/RefreshFM14.zip

Please submit any bug reports or issues to the support site.

Upgrades

New licences are $499 and upgrades from version 3 or earlier are $199. Existing licence holders will be emailed a discount code to unlock the upgrade pricing. If you own a copy of an older version of RefreshFM and would like to upgrade, please contact us for your code.

RESTfm goes Open Source - RESTful Web Services for FileMaker Server now free

We released the first version of RESTfm back in 2012 and although we talk about it at every DevCon, it's our least understood and used product. Developers at the conference can easily see the benefits of a tool like BaseElements and they understand the concepts of RefreshFM.

But only a minority of FileMaker developers we meet would also work in Web Services. So RESTfm as a paid product only appeals to those who already know what they're looking for. But RESTfm is so much more. We use it to :

  • Deliver solutions to iOS.
  • Register users via one time login codes.
  • Post invoices written in the field to the accounts department.
  • Allow a Siebel solution to talk to FileMaker Server via BizTalk.
  • Run a complete syncing product on iOS and Desktop.
  • Deliver FileMaker data to a native iOS app.

Web Services makes these sorts of things possible.

RESTfm makes it not only possible, but easy to do from your existing FileMaker solutions. And we want more people to be able to try it out and see this for themselves.

So we go Open Source

So we've turned RESTfm from a paid only product into an Open Source model and added a paid annual sponsorship instead. We'd like all FileMaker developers to try out RESTfm and see what sorts of things they can do with Web Services in their FileMaker Server data.

The RESTfm code is available now at our GitHub repository, under an MIT Licence :

https://github.com/GoyaPtyLtd/RESTfm

and can be downloaded here :

https://github.com/GoyaPtyLtd/RESTfm/releases/download/v3.4.0/RESTfm-3.4.0.zip

Support packages include our free syncing framework, a set of sample files and access to a paid only support option. Sponsorship is $199 for individuals or $399 for companies of more than one developer.

http://restfm.com/purchase

We've already got plans to continue growing both RESTfm and our sample code and sync framework, and we hope that by making RESTfm Open Source, more people can see the benefits that we've already taken advantage of.

BaseElements 5 preview - Part 1 - History

The world of software doesn't stand still and FileMaker Pro is no exception. All through the FileMaker releases we've kept BaseElements up to date with each new release, and almost always fairly soon after release.

In working on a new UI for BaseElements 5 we've had to re-consider every element in the product. In doing that, we've looked back at what we've introduced so far, and what worked, what didn't and where we can improve things.

A little history - BaseElements 1

BaseElements 1 was actually built in FileMaker Pro 6. Yes the same FileMaker Pro 6 of the one table per file. Although the final release version was a FileMaker 7 file, it still had a bit of a hangover from it's starting point, as there were multiple files for different parts of the UI.

BE1_0.png

I did really like the icons I found at the time, it was a set called "Harmony" from IconFactory, but not everyone agreed.

The first release was in January of 2007 after a couple of months of private and public beta testing, and at the time, FileMaker was at version 8. My favourite FileMaker fp5 developer tool, MetaDataMagic hadn't yet been released for FileMaker 7 files.

BaseElements got it's first big show at DevCon that year, in Orlando Florida, after being updated earlier ( BE 1.5.0 on 10th July 2007 ) for the just released FileMaker Pro 9.

Side note about taking a product to DevCon for the first time.

I'd been to DevCon once before, but not as an exhibitor. I'd never done an exhibitor or trade show booth before in my life. I was thinking I'd need to have some sort of gimmick to attract crowds, like maybe play on the Aussie background and make people Vegemite toast.

After checking into my room, I got into the elevator with none other than Danny Mack - developer at NMCI who made MetaDataMagic. He mentioned that they were working on a new release for fp7 files. I had a moment of thinking that all my work would be undone if they came out with something good. I worried : will people buy my new untested product, when there's a update to a well established one just around the corner? I had a feeling of dread, thinking he might be going to announce something this week.

When the conference opened, I was standing behind my desk, a small sign behind me and a monitor in front with BE running. Someone walked up, looked at my sign and walked off. That didn't go well, so I think : "Ok, fine, I'll have to get out and spruik it a bit". So I step out from behind the desk.

Another visitor walks up, looks at the sign. I ask : "Can I show you BaseElements?". "No thanks" and he walks off. I have the feeling of dread for the second time, and thought it was going to be a really long four days. What had I signed myself up for? I'm not a natural salesman, more of an introvert, so this whole exercise was pushing back against my nature.

A few minutes go by, another person walks up. "Can I show you BaseElements?". "Yeah sure". So I do the demo, and mention the discount, and before I can say anything else, he pulls out a credit card and I have a sale. I was elated.

From that moment on I was hectic most of the time. People would buy a copy, and then bring others back to show it to friends or colleagues. I was doing credit cards manually on the website and it took ages, and I'd often have people looking over BaseElements themselves while I had a queue of people waiting to buy a copy.

It was a mad four days, and I barely ate a meal I was so busy, occasionally someone would bring me a beer. But I wouldn't have swapped it for anything.

We had some more tweaks later in the year, including our comparison reports and lots of additional features, but no new FileMaker releases until...

BaseElements 2 - consolidations

BaseElements 2 came out just before DevCon 2008. FileMaker had introduced Autosizing in FileMaker 9, and we redid the interface, as well as consolidating it into a single UI file.

BE2_0.png

The fun part of building BaseElements 2 was using BaseElements to consolidate BaseElements, by writing a consolidation report ... in BaseElements. It was an interesting experience.

The big design change was in the left side navigation for Form view. Which is great, except it didn't collapse or expand automatically, and it didn't appear on any of the list or table layouts.

I'd bet that every FileMaker developer would love a native UI component that allowed you to build a side navigation, but one that appeared in every type of layout view. Strangely this idea must be valuable, as it kept coming back in the designs ideas for BaseElements 5 ( more of that later ).

Generally though, the design of BE 2 remained the same, although the interface itself got a bit cleaner ( fewer ID fields on the layouts ), it also added a bunch of extra features and UI elements ( Link layouts ) so the net effect was probably neutral.

January 2009 saw the release of FMP 10, and we had a minor update to BE 2 out the door almost the same day. We continued for the rest of that year to extend and add features.

FileMaker Pro 11 came out in early 2010. There was an update to the way FileMaker's internal XSLT engine worked, which meant that BE 2 wouldn't import properly in FMP 11. So we put a lot of work into our import engine, and also sped it up, just in time for DevCon 2010...

BaseElements 3 - ditch the side toolbar

So in spite of the fact that the left side toolbar came into the product only in v2, by v3 it was already gone. Version 3 was the last version to have a big design change to the UI. We consolidated all the toolbars to a single one at the top of the layout ( minus my favourite icons ) and continued to simplify the layouts.

BE3_0.png

And although some things were taken away ( link layouts ), some other things were added that took up more UI. We had a semi hidden UI for accessing Notes, and a completely hidden UI for accessing Copy to Clipboard. We added a Smart Find feature, which people loved, and a coloured note tag option which no-one cared about.

It wasn't long into 2012 that FileMaker released version 12. This was a new file format, and lots of new features, but there wasn't much change in the DDR so we had a free update to BE 3 on the same day. But there was no big new change to the UI to accompany it.

FMP 12 enabled themes which began to give us much more control over the UI and make UI design much easier in the long run. In the short term, for existing projects though, it meant you had to re-do everything on the layout. So we decided to persist a bit more with what we had for the next release.

BaseElements 4 - more functionality, same UI

So even though we considered re-doing the interface for BaseElements 4, in the end it didn't get done. We'd worked on a bunch of new features but the interface remained almost exactly the same.

We also experimented with some ideas : a highlight function which I doubt many ever used. There was a SQL conversion item in the Tables which I know one developer absolutely loved ( it was the starting process for an idea to have the data store for BE in a MySQL table using ESS instead of in FMP - that's still in the ideas list ).

Our invisible Copy to Clipboard interface was extended to Tables and Scripts, we added the ability to mark errors as fixed with an additional checkbox on every form view. And as an homage to the grand functionality enabled by themes, we added a rollover effect to links, which only served to make it harder to know what a link was and what wasn't without first rolling over them. There was a highlight option no one asked for, but I thought might be useful, and never used.

We continued to tweak things and updated again for FileMaker Pro 13 compatibility when that was released in February 2014, and again for FileMaker Pro 14 in June this year. BE 4 was the only version to span three whole FileMaker versions.

BaseElements has grown organically, some of the time through feature requests from users, in other places from ideas of mine, and sometimes just going where the code takes you. But in all these releases we never went back and did a wholesale review of what was there and why, and what worked and what didn't.

But the last two years we've been re-thinking what BE should be, and how it should work, and from that, what it should look like.

We're going to have a beta release out in time for DevCon this year, and here's a preview :

BE5_0.png

In part 2 of our BaseElements 5 preview I'm going to talk about the vision we have for the BaseElements product, and the thought process that's gone into this redesign.

BaseElements Plugin 3.1 beta available, now includes SMTP

Hot on the heels of yesterday's note about the 3.0 version of our free FileMaker plugin, we've got a new beta release available already. This version does some cool new stuff :

  • Support for Gzip and UnGzip which is the same format as the internal FileMaker container field compression.
  • Support for testing container fields for compression.
  • Import into container fields, to match the Export so that you can do this inside FMS scripts.
  • New CURL constants to simplify HTTP Authentication.
  • Additional options for the Values functions to support case insensitivity.
  • The ability to use callback functions to call scripts, or evaluate calculations from with JavaScript.

And ... new SMTP email sending functionality, with support for HTML emails and multiple attachments. We knew you'd like that last one :)

Beta releases are available to all of our sponsors, you can subscribe for just $199 for single developer, and $399 for a company. Our sponsors help to decide on future functionality and get access to our support forums.

Syndicate content