Goya Blog

BaseElements Update and more

It's been a while since the last BaseElements update, and I've had a few emails asking me what's going on. Rest assured we're still working on it, and things are progressing.

And FileMaker 15 is out today, and happens to coincide with me finishing up a big set of changes to BE, so an opportune time to post an update. So to answer all the big questions :

Are you still working on it?

Absolutely. We still use BE every day ourselves, and even use BE to develop BE and couldn't live with out it, much like a lot of you. We haven't sat still but development has been slower while we've worked on client work, RefreshFM updates, RESTfm updates and additions and bug fixes to the Plugin.

Why is importing objects so slow in b12?

What's mostly got us stuck is some changes to the Alerts that we introduced with the new 14 re-design. In order to better track the myriad of issues that can crop up, and be better able to handle errors, warnings, performance issues and unreferenced items, we've been consolidating them into a single alert. So we've added some extra items and included more detail for others, plus combined things into a single location.

And as every developer knows, when you add to a solution iteratively over a long period, all your changes add up. And what we ended up with was a solution for layout objects that was way too complex than it should be to get the job done. At some point in the last few betas our alerts auto enter calculation was doing a sum of a related field, which itself was un-stored, and that will never be fast.

You never set out to design a slow solution, and you never deliberately make a change that is going make things slow, but occasionally you add a great feature that has a cost you didn't plan for.

The solution was to go back and re-think the whole way alerts are generated across all the elements, but especially the large ones ( objects, steps, calculations ). And to do that meant putting a lot of these calculations into the XSLT so that the import was as simple as possible, and therefore as fast as possible.

If you want fast imports, remove all your unnecessary auto enters, lookups, stored calculations and indexes.

We've spent a lot of time re-factoring our XSLT to shift a lot of the complexity there instead of the import. This is a complex, difficult process - XSLT is handled differently in the plugin, vs the command line, vs the import, vs the debugger tools. This makes it really time consuming to debug and test any change.

Our base line for speed testing was importing our sample files ( BaseElements itself ) into the current BE 4. That takes about 5 minutes. We knew that the latest beta of 14 was slower, quite a bit slower. We needed to at least get back to where things were with that version.

At various times during the changes we had imports that took one minute, only to realised hardly any data had come in, and 7 minutes only to make a simple namespace change that cut that in half.

Finally we've got all our data, all working as we want and import times are ... drum roll ... 3 minutes. About a 40% savings on BE 4 and much faster than BE14b12. Excellent, not only did we fix the b12 issue, but we also improved on the previous version.

What's with the beta now that 15 is out?

Our plan with BE 14 was that we'd finish the beta late last year, and then release BE 15 on the release of FMP 15. Obviously that didn't go to plan.

We're fortunate that there are almost no changes to the DDR with 15 so our current release version of BE 4 will work with the FMPA 15 DDR just fine. So will the BE 14 beta.

So we're going to re-start the BE versioning at 15 and go back to b1 and pretend that the whole BE 14 beta never happened :) Well not quite, but it doesn't make sense to have BE 14 around with FMP 15.

There is no upgrade cost from BE 14 to 15. So we will now work on finishing up the remaining features we were working on ( reports mostly ) and release BE 15 as what we were going to do with 14. Everyone who has bought BE back since mid last year will get the final release of BE 15 at no charge and all the current BE 14 licences will work just fine with BE 15.

What else have you done?

One of the biggest support issues we have is in bad XML that FMPA generates that is invalid and so can't be imported. We are pre-processing a lot of that and removing some of these characters and so a lot of imports that would have failed in older versions will work in BE 15.

What else are you going to do?

We've still got some tweaks to how the UI works in BE 15 that we need to work on. There is the reports to do, and we've got some great ideas about those that we think are going to be awesome.

Thanks to everyone for your patience, and we hope it's not going to be much longer till the BE 15 final. In the mean time grab a copy, and send us any feedback.

BaseElements beta 3

We've posted a new version of the BaseElements 14 beta, the new download is b3. Although I was hoping to have a final release done by now, it's taken a bit longer than anticipated.

This version fixes the windows import issue that quite a few people saw, so I've put back the mac and win runtimes. There is a beta version of the plugin out for this.

Great new addition to this version is a new version of fmSearchResults. This is a beta release, with a heap of new functionality, so I'll post more about it soon, but it's good to have this back.

Hopefully this week will finish up some of the outstanding issues - reports, and some of the error notifications, and then we should be right to wind up the beta and start looking towards new functionality.

BaseElements Plugin Updated to 3.1 adds SMTP and more

And one more release note, we've updated the free BaseElements Plugin to 3.1.0. This is a huge release, lots of cool new features.

This version removes support for Windows XP, so be aware of that if you're running older OS versions.

But in return you get SMTP along with JPEG compression for reducing container picture sizes. There's a ConvertContainer for turning a file to a picture and vice versa, so no more exporting and re-importing. There's the ability to evaluate JavaScript inside the plugin, which opens a whole world of possibilities. And BE_ImportFile for importing files into containers - you can do this in the client, but now you can also do it on the server via the plugin.

The documentation will be updated in the next few days, I'll post another note here when it's done.

Sponsors

Thanks again to all the people and companies that sponsor the plugin. It's because of their support that you get all this for free, so if you use the plugin and know someone who sponsors, make sure to buy them a round.

And if you can, please consider sponsoring. It doesn't cover all our costs, but it does help, and you get a proper support channel and a bunch of other cool plugin related code.

New Functions

  • BE_ContainerIsCompressed
  • BE_ConvertContainer
  • BE_Curl_Trace
  • BE_EvaluateJavaScript
  • BE_Gzip
  • BE_ImportFile
  • BE_JPEG_Recompress
  • BE_SMTP_Send
  • BE_SMTP_Server
  • BE_UnGzip
  • BE_Values_ContainsDuplicates
  • BE_Xero_GenerateKeys

Other Changes

  • BE_Curl_Set_Option: label function parameters as optional
  • BE_Curl_Set_Option: the options CURLOPT_HTTPAUTH & CURLOPT_PROXYAUTH take named constants
  • BE_HMAC: add a parameter to allow and input encoding for Base64 and Hex
  • BE_HTTP_POST: allow post args to specify file paths using the @ syntax
  • BE_HTTP_POST: do not crash when a http post contains a parameter that is not a key=value pair
  • BE_JSON_Encode: correctly handle leading 0 for real numbers
  • BE_Values_Unique, BE_Values_FilterOut: add option to be case insensitive
  • Ensure that Get ( InstalledFMPlugins ) returns the same value on both OS X and Windows
  • Remove support for Windows XP

BaseElements 14 beta now available

BaseElements has grown up so quickly it's skipped 9 whole version numbers. Straight from version 4 to version 14, in order to better keep track of compatibility with the current release of FileMaker Pro.

We have a beta version available :

http://www.goya.com.au/files/baseelements/14.0.0b1/BaseElements14_Mac_Runtime.zip
http://www.goya.com.au/files/baseelements/14.0.0b1/BaseElements_14.zip

I started talking about the design of BaseElements in an earlier blog post, and I'll have a lot more to say on that, including a followup about our reasons for making the changes we have.

But for now I wanted to get this beta out so people at DevCon can have a play and see for themselves.

Upgrades

This is a paid upgrade, you can purchase a discounted upgrade during DevCon but the discounts won't last, so get in quick.

RefreshFM 14 beta now available

We've uploaded a test version of RefreshFM ready for FileMaker 14. The only changes in this release are in the automated setup, which now works in FileMaker 14 and 13.

This is a paid update, and you may have noticed that we skipped a few versions, going from 3 to 14. We decided it was better to match the FileMaker Pro version directly so it's clear what release of FileMaker is required.

Download

You can download the new version here :

http://www.goya.com.au/files/rfm/14.0.0b1/RefreshFM14.zip

Please submit any bug reports or issues to the support site.

Upgrades

New licences are $499 and upgrades from version 3 or earlier are $199. Existing licence holders will be emailed a discount code to unlock the upgrade pricing. If you own a copy of an older version of RefreshFM and would like to upgrade, please contact us for your code.

RESTfm goes Open Source - RESTful Web Services for FileMaker Server now free

We released the first version of RESTfm back in 2012 and although we talk about it at every DevCon, it's our least understood and used product. Developers at the conference can easily see the benefits of a tool like BaseElements and they understand the concepts of RefreshFM.

But only a minority of FileMaker developers we meet would also work in Web Services. So RESTfm as a paid product only appeals to those who already know what they're looking for. But RESTfm is so much more. We use it to :

  • Deliver solutions to iOS.
  • Register users via one time login codes.
  • Post invoices written in the field to the accounts department.
  • Allow a Siebel solution to talk to FileMaker Server via BizTalk.
  • Run a complete syncing product on iOS and Desktop.
  • Deliver FileMaker data to a native iOS app.

Web Services makes these sorts of things possible.

RESTfm makes it not only possible, but easy to do from your existing FileMaker solutions. And we want more people to be able to try it out and see this for themselves.

So we go Open Source

So we've turned RESTfm from a paid only product into an Open Source model and added a paid annual sponsorship instead. We'd like all FileMaker developers to try out RESTfm and see what sorts of things they can do with Web Services in their FileMaker Server data.

The RESTfm code is available now at our GitHub repository, under an MIT Licence :

https://github.com/GoyaPtyLtd/RESTfm

and can be downloaded here :

https://github.com/GoyaPtyLtd/RESTfm/releases/download/v3.4.0/RESTfm-3.4.0.zip

Support packages include our free syncing framework, a set of sample files and access to a paid only support option. Sponsorship is $199 for individuals or $399 for companies of more than one developer.

http://restfm.com/purchase

We've already got plans to continue growing both RESTfm and our sample code and sync framework, and we hope that by making RESTfm Open Source, more people can see the benefits that we've already taken advantage of.

BaseElements 5 preview - Part 1 - History

The world of software doesn't stand still and FileMaker Pro is no exception. All through the FileMaker releases we've kept BaseElements up to date with each new release, and almost always fairly soon after release.

In working on a new UI for BaseElements 5 we've had to re-consider every element in the product. In doing that, we've looked back at what we've introduced so far, and what worked, what didn't and where we can improve things.

A little history - BaseElements 1

BaseElements 1 was actually built in FileMaker Pro 6. Yes the same FileMaker Pro 6 of the one table per file. Although the final release version was a FileMaker 7 file, it still had a bit of a hangover from it's starting point, as there were multiple files for different parts of the UI.

BE1_0.png

I did really like the icons I found at the time, it was a set called "Harmony" from IconFactory, but not everyone agreed.

The first release was in January of 2007 after a couple of months of private and public beta testing, and at the time, FileMaker was at version 8. My favourite FileMaker fp5 developer tool, MetaDataMagic hadn't yet been released for FileMaker 7 files.

BaseElements got it's first big show at DevCon that year, in Orlando Florida, after being updated earlier ( BE 1.5.0 on 10th July 2007 ) for the just released FileMaker Pro 9.

Side note about taking a product to DevCon for the first time.

I'd been to DevCon once before, but not as an exhibitor. I'd never done an exhibitor or trade show booth before in my life. I was thinking I'd need to have some sort of gimmick to attract crowds, like maybe play on the Aussie background and make people Vegemite toast.

After checking into my room, I got into the elevator with none other than Danny Mack - developer at NMCI who made MetaDataMagic. He mentioned that they were working on a new release for fp7 files. I had a moment of thinking that all my work would be undone if they came out with something good. I worried : will people buy my new untested product, when there's a update to a well established one just around the corner? I had a feeling of dread, thinking he might be going to announce something this week.

When the conference opened, I was standing behind my desk, a small sign behind me and a monitor in front with BE running. Someone walked up, looked at my sign and walked off. That didn't go well, so I think : "Ok, fine, I'll have to get out and spruik it a bit". So I step out from behind the desk.

Another visitor walks up, looks at the sign. I ask : "Can I show you BaseElements?". "No thanks" and he walks off. I have the feeling of dread for the second time, and thought it was going to be a really long four days. What had I signed myself up for? I'm not a natural salesman, more of an introvert, so this whole exercise was pushing back against my nature.

A few minutes go by, another person walks up. "Can I show you BaseElements?". "Yeah sure". So I do the demo, and mention the discount, and before I can say anything else, he pulls out a credit card and I have a sale. I was elated.

From that moment on I was hectic most of the time. People would buy a copy, and then bring others back to show it to friends or colleagues. I was doing credit cards manually on the website and it took ages, and I'd often have people looking over BaseElements themselves while I had a queue of people waiting to buy a copy.

It was a mad four days, and I barely ate a meal I was so busy, occasionally someone would bring me a beer. But I wouldn't have swapped it for anything.

We had some more tweaks later in the year, including our comparison reports and lots of additional features, but no new FileMaker releases until...

BaseElements 2 - consolidations

BaseElements 2 came out just before DevCon 2008. FileMaker had introduced Autosizing in FileMaker 9, and we redid the interface, as well as consolidating it into a single UI file.

BE2_0.png

The fun part of building BaseElements 2 was using BaseElements to consolidate BaseElements, by writing a consolidation report ... in BaseElements. It was an interesting experience.

The big design change was in the left side navigation for Form view. Which is great, except it didn't collapse or expand automatically, and it didn't appear on any of the list or table layouts.

I'd bet that every FileMaker developer would love a native UI component that allowed you to build a side navigation, but one that appeared in every type of layout view. Strangely this idea must be valuable, as it kept coming back in the designs ideas for BaseElements 5 ( more of that later ).

Generally though, the design of BE 2 remained the same, although the interface itself got a bit cleaner ( fewer ID fields on the layouts ), it also added a bunch of extra features and UI elements ( Link layouts ) so the net effect was probably neutral.

January 2009 saw the release of FMP 10, and we had a minor update to BE 2 out the door almost the same day. We continued for the rest of that year to extend and add features.

FileMaker Pro 11 came out in early 2010. There was an update to the way FileMaker's internal XSLT engine worked, which meant that BE 2 wouldn't import properly in FMP 11. So we put a lot of work into our import engine, and also sped it up, just in time for DevCon 2010...

BaseElements 3 - ditch the side toolbar

So in spite of the fact that the left side toolbar came into the product only in v2, by v3 it was already gone. Version 3 was the last version to have a big design change to the UI. We consolidated all the toolbars to a single one at the top of the layout ( minus my favourite icons ) and continued to simplify the layouts.

BE3_0.png

And although some things were taken away ( link layouts ), some other things were added that took up more UI. We had a semi hidden UI for accessing Notes, and a completely hidden UI for accessing Copy to Clipboard. We added a Smart Find feature, which people loved, and a coloured note tag option which no-one cared about.

It wasn't long into 2012 that FileMaker released version 12. This was a new file format, and lots of new features, but there wasn't much change in the DDR so we had a free update to BE 3 on the same day. But there was no big new change to the UI to accompany it.

FMP 12 enabled themes which began to give us much more control over the UI and make UI design much easier in the long run. In the short term, for existing projects though, it meant you had to re-do everything on the layout. So we decided to persist a bit more with what we had for the next release.

BaseElements 4 - more functionality, same UI

So even though we considered re-doing the interface for BaseElements 4, in the end it didn't get done. We'd worked on a bunch of new features but the interface remained almost exactly the same.

We also experimented with some ideas : a highlight function which I doubt many ever used. There was a SQL conversion item in the Tables which I know one developer absolutely loved ( it was the starting process for an idea to have the data store for BE in a MySQL table using ESS instead of in FMP - that's still in the ideas list ).

Our invisible Copy to Clipboard interface was extended to Tables and Scripts, we added the ability to mark errors as fixed with an additional checkbox on every form view. And as an homage to the grand functionality enabled by themes, we added a rollover effect to links, which only served to make it harder to know what a link was and what wasn't without first rolling over them. There was a highlight option no one asked for, but I thought might be useful, and never used.

We continued to tweak things and updated again for FileMaker Pro 13 compatibility when that was released in February 2014, and again for FileMaker Pro 14 in June this year. BE 4 was the only version to span three whole FileMaker versions.

BaseElements has grown organically, some of the time through feature requests from users, in other places from ideas of mine, and sometimes just going where the code takes you. But in all these releases we never went back and did a wholesale review of what was there and why, and what worked and what didn't.

But the last two years we've been re-thinking what BE should be, and how it should work, and from that, what it should look like.

We're going to have a beta release out in time for DevCon this year, and here's a preview :

BE5_0.png

In part 2 of our BaseElements 5 preview I'm going to talk about the vision we have for the BaseElements product, and the thought process that's gone into this redesign.

BaseElements Plugin 3.1 beta available, now includes SMTP

Hot on the heels of yesterday's note about the 3.0 version of our free FileMaker plugin, we've got a new beta release available already. This version does some cool new stuff :

  • Support for Gzip and UnGzip which is the same format as the internal FileMaker container field compression.
  • Support for testing container fields for compression.
  • Import into container fields, to match the Export so that you can do this inside FMS scripts.
  • New CURL constants to simplify HTTP Authentication.
  • Additional options for the Values functions to support case insensitivity.
  • The ability to use callback functions to call scripts, or evaluate calculations from with JavaScript.

And ... new SMTP email sending functionality, with support for HTML emails and multiple attachments. We knew you'd like that last one :)

Beta releases are available to all of our sponsors, you can subscribe for just $199 for single developer, and $399 for a company. Our sponsors help to decide on future functionality and get access to our support forums.

BaseElements Plugin Updated to version 3.0

We've released another version of our free FileMaker plugin, this time to 3.0. This versions brings some new functionality :

  • BE_ExportFieldContents - allowing you to do exports from containers on FMS
  • BE_FTP_Upload - self explanatory I hope.
  • BE_HMAC - encoding format often used in Web Services authentication
  • BE_Xero_SetTokens - to allow the use of the Xero API from within FileMaker.

This also adds some new, but hidden functionality, that will be finalised in a future release ( the soon to arrive 3.1 version )

  • BE_EvaluateJavaScript
  • BE_SMTP_Server
  • BE_SMTP_Send

BE_EvaluateJavaScript looks to be very interesting and have a host of neat possibilities.

This version removes some previously deprecated functions :

  • BE_ExecuteShellCommand
  • BE_FileMaker_Tables
  • BE_FileMaker_Fields

If you were relying on BE_ExecuteShellCommand you will need to change your code to use BE_ExecuteSystemCommand instead, and the two internal SQL functions can be done with the new Execute SQL function.

As well there's been error handling fixes to BE_ProgressDialog, the BE_ListFilesInFolder will force the correct path separator, BE_MessageDigest will show an error for passing an invalid algorithm and the Zip functions have better error handling and now support empty folders inside zip files.

If you appreciate the plugin, and use it in your solutions, please consider an annual sponsorship. Just $199 for single developer, and $399 for a company. Sponsorship gives you access to technical support, new releases, code samples and lets you help decide on future functionality.

BaseElements Plugin Future Directions

You may have noticed a new version of our free FileMaker plugin was released in the last few weeks, bringing the version number to 3.0. Lots of new features in this that are very exciting, but there's been a lot going on behind the scenes here as well which hasn't been public and I'd like to go through for all those interested.

It started at Xero

One of the big new things we added in this release is a simple ( on the outside ) function to authenticate against the Xero API. Xero is becoming one of the big new things for online accounting, we use it ourselves and it's growing worldwide.

At first it seemed like we would need OAuth in the plugin to make this work, so we started down that path. It turns out the Xero interaction is OAuth like but not the same as what we'd developed. After a lot of back and forth and development work at our end, we finally got something working, but realised the function itself is just the beginning.

So we also set to work to write a Xero sync framework that can push and pull data from Xero to a set of local FileMaker tables. This was something beyond the plugin, but that really only has value alongside the plugin.

At this point, we had the plugin as a set of functions that were free, open source and contributed from lots of different people, and a second set that we'd spent way more time and money on, and that also requires a FileMaker sync framework to be useful, that in our mind could be a product on it's own.

We spent a fair bit of time looking at how we could have two versions of the plugin, one paid and and one free. The paid version would just contain the free functions plus extras. Seems simple enough, even with our model of not having registration functions, until you get into real world questions. For example you want to be able to update from the free to the paid. So they have to have the same names and function names. But then how do you know which is which?

It quickly became apparent that this would only create more work for the people who have paid, who we are trying to improve things for. And the whole point of this at the beginning was to have a single plugin, with no complex registration.

So all of that has been shelved, and we're back to a single set of open source code, but we need to consider now what to do with the Xero work and then whether we can recover any of the costs of development to date.

Acknowledging the work done so far

All of the plugin development has been done by an outside contractor, so everything we do has a cost. Even though we've had some very generous people either contribute code or pay for changes to code, that has been less than one tenth of the total spend to date.

It doesn't stop here

But we've also got a lot planned for the plugin already, including some cool javascript code, and support for SMTP. But on top of that, we are now seeing much more in the way of support requests from people who need implementation help.

Also there have been requests for functionality that we just can't consider, or that would be a lot of work, but that aren't going to be very popular. It would be nice to have a way to get feedback from users about what should go into the plugin, and how we should extend it.

The Future

So we've decided give people a better option to sponsor the plugin in order to have a say in where it goes in the future, and to help contribute towards it's ongoing costs. This would be an annual fee ( $199 for a single user or one person company, and $399 for a larger organisation ). There is no obligation to take this up, you still don't need to register the plugin, we will continue to open source all of the code, and there will not be a paid version that is separate from the free one.

But anyone who is a sponsor will have access to :

  • A dropbox share of every released version of the plugin with instant updates.
  • Access to test versions of new releases.
  • Any plugin dependent FileMaker code, starting with the Xero code, along with a licence to integrate it in any of their own projects for free, again with no registration.
  • Access to a proper support channel.
  • A say in what functions will be added in the future, and the ability to suggest new functionality.
  • A warm fuzzy feeling that you're supporting something that all FileMaker developers can benefit from.

Great idea, let me buy

If you're super keen to get started, you can subscribe here :

Single developer subscription - $199

Company subscription - $399

There is the option to create an account and password, we suggest you do that if you want to manage any future transactions. The payment gateway is BlueSnap, so you'll get confirmation details from them.

We won't be creating a NFP or EDU discount on this, but will let larger organisations purchase a single developer subscription.

Can I tempt you with more?

We've just released the 3.0 version of the plugin, which adds Exporting of containers on FMS, FTP Uploads, Xero auth and more. And 3.1 is just around the corner and it does JavaScript and SMTP.

Thanks

Thank-you to everyone who has contributed in the past, either code or sponsoring functions. Thanks in advance to everyone who values the plugin enough to sponsor us.

Syndicate content