Externally referencing project resources

[sarahb]

Hi all,

I currently have one master Flare project that includes our company's training for multiple training projects. Originally, we decided to put all the training together in one project because they all use the same project resources: CSS, page layouts, snippets, etc. It was just easier to update the resource in one project, rather then copy the update into 20 individual projects. However, the master project is starting to get very unruly, and the cons are starting to outweigh the pros.

So my question is: are there any project resources that can live in one place outside of the Flare project that all Flare project's can externally reference (i.e. link to remotely)? We are using Git as source control if that matters, and have the latest Flare release (currently Flare 2019 R2).

Thanks in advance!

[doloremipsum]

There are a couple of ways you can do that:

- External Resources, which let you map a folder in Flare to an external folder and synchronise changes between them. See here in the Help: https://help.madcapsoftware.com/flare20 ... ources.htm

- Global Project Linking, which lets you set up a project that contains Global resources and import them into other Flare projects. See here: https://help.madcapsoftware.com/flare20 ... inking.htm

They both have their pros and cons, so I suggest that you read up on them and come back with any specific questions. Ultimately it will depend on your particular situation (what kind of files are shared, how often you need to edit, how many collaborators you have, how good they are at synchronising regularly, etc)

I'll give you one tip for Source Control: strongly recommend that you only source control any file in one place. e.g. if you have Global Project linking, those files should be source controlled ONLY in the Global project, and be excluded from source control in the child projects. That way, even though there are a number of copies of the same file floating around, one of them (the one in the remote) is the source of truth and any changes have to go through there.

[NorthEast]

I'll give you one tip for Source Control: strongly recommend that you only source control any file in one place. e.g. if you have Global Project linking, those files should be source controlled ONLY in the Global project, and be excluded from source control in the child projects. That way, even though there are a number of copies of the same file floating around, one of them (the one in the remote) is the source of truth and any changes have to go through there.

I can see why you might do that, but it's not something I'd recommend myself - or at least you need to be aware of the implications.

If you don't include these files in source control in the child projects, two people could quite easily have different versions of these files. That's because it depends entirely on how that person imports the files, and when they last imported the files. Therefore, two people could produce outputs that are different, whilst thinking they're using the "same source". For my projects, what we have in source control includes everything, so people can reliably build the same output on different PCs.

I use project imports (GPL) with master and child projects and, yes, there are problems when you use source control - but that's due to how the Flare import works. Flare's project import checks to see what files are new and need reimporting, but it does this by checking the file's timestamp on your local PC - it is not checking the version of the file in source control. As the file's timestamp on your local PC depends on when you last got that file from source control, you can get in scenarios where you can update files in the master (source) project, but these aren't reimported because the files in the local (child) project have a newer date - despite these local files being an "older" version of the file (in source control terms).

There's only one way I've found to reliably solve this problem - whenever you run a project import, you need to reimport all files that have changed - both Source is Newer and Local is Newer.

[doloremipsum]

It's probably six of one, half a dozen of the other really. We keep all of our (many many) snippets in the global project, so the list is really too long to look through for every import (and it takes too long to over-write the whole lot). The easier workflow for us is Synchronise Global -> Import into Child ->Build.

[ChoccieMuffin]

We have the same approach as doloremipsum, but we don't have the problem with different outputs you describe, Dave, because our builds happen on a separate build machine whose starting point is to check out everything from SVN and start from there, and our import files in SVN are always kept "clean", so the build machine starts off with a blank sheet. Though I appreciate not everyone is able to work like this.

All global stuff (stylesheets, page layouts, other assorted content, TOCs and condition tag sets and variable sets that go into every project) lives in our globals project, and we have imports from one child project into another in several instances, and it's pretty stable now and works well for us, as we - a team of two - clearly understand that any outputs we build on our own machines are never going to be the final deliverables.

When you do decide which approach to take, what really matters is everyone on the team using the same process properly, and that's not something that your Flare setup can control.

[NorthEast]

We have the same approach as doloremipsum, but we don't have the problem with different outputs you describe, Dave, because our builds happen on a separate build machine whose starting point is to check out everything from SVN and start from there, and our import files in SVN are always kept "clean", so the build machine starts off with a blank sheet. Though I appreciate not everyone is able to work like this.

So you don't put all the files in your project in source control? You leave out the files that came from an import?

You'll have the same potential problem because if you don't include all files, then you don't have a complete version of your project in source control. There's nothing to ensure that the files that aren't included are going to be the same on each PC.

Fair enough that if you only ever build from one PC then it'll be consistent, and you won't notice. But the build on your PC can and might be different to what your build PC produces.

[ChoccieMuffin]

We have the same approach as doloremipsum, but we don't have the problem with different outputs you describe, Dave, because our builds happen on a separate build machine whose starting point is to check out everything from SVN and start from there, and our import files in SVN are always kept "clean", so the build machine starts off with a blank sheet. Though I appreciate not everyone is able to work like this.

So you don't put all the files in your project in source control? You leave out the files that came from an import?

You'll have the same potential problem because if you don't include all files, then you don't have a complete version of your project in source control. There's nothing to ensure that the files that aren't included are going to be the same on each PC.

Fair enough that if you only ever build from one PC then it'll be consistent, and you won't notice. But the build on your PC can and might be different to what your build PC produces.

Yes, correct, we leave out imported files in the source control for individual projects. We get over this by having - and using - a process that is defined outside of the SVN process, which says that if you want to build stuff locally you have to update all the projects in the "Docs" area first. It works for us, and builds that we do locally aren't for public consumption anyway, just for us to use when checking what we've done. So if we come across an oops, the first thing we check is that we've done that first step and updated all our projects before building. (It has the benefit of preventing our build manager from complaining that we're filling the SVN repository with duplicated stuff, so one less moan in our ears!)

[techwriter31]

We have taken a different approach, where we use SVN externals in our local projects, and these map to the our "Global" project. This allows us to bypass the Flare manual import completely. The manual import within Flare was always a problem because:
-The timestamp issue that Dave mentioned.
-We didn't want writers to have to create a working copy of the global project, update it, and then perform the import process in their local project. So, we replicated the global project to a network location. However, we have writers in various locations, and the import process could take a very long time for anyone remote.
-We did have our imported files under version control in the local project, so writers had to get a lock on all of the imported files so they were read/write. If ANY file was read-only, the import would silently fail.
-Writers could modify this imported global content in the local working copy and commit the change. Eventually it would be sorted out, but there was nothing preventing it.

Now that we're using SVN externals:
-Each local/child project is mapped to the global project. Writers simply update their local working copy to get the latest global content.
-The externalized files cannot be locked or committed. So even if a writer makes a change to these global files in their local project, it never gets committed to version control.
-We use TeamCity for continuous integration and it fully supports SVN externals. So anytime we commit changes to our global content, the documentation is automatically rebuilt with these changes in place.

[sarahb]

OP here. Thanks all very much. I went with the global project linking option and it has worked very well in all my tests.

My next question is whether it's possible to import the global project from a remote server, specifically Bitbucket (Git)? Both the global "parent" project and the "child" projects will be located on Bitbucket. Right now, since I'm in the testing phase, both the global "parent" project and the "child" project are local and not connected to source control.

Following MadCap's documentation, to import Flare files from the global "parent" project, I'm doing the following in my "child" project: Click Project > Import > Flare Project, and in the Project file field, select the global "parent" project. I've also set it to automatically sync with the global "parent" project each time output is generated. As I mentioned, currently these projects are located locally and not connected to source control. However, I don't see an option to be able to import a Flare project from a remote server--it only lets me browse my computer. I'm not sure if options change once the project is linked to source control, or whether importing the global project from a remote server is not possible.

Also, providing it is possible to import the global "parent" project from a remote server, would each individual on my team need to have the global project checked out locally, or can they just have their assigned child project checked out, and that child project imports the global project living on the remote server?

Sorry if any of that wasn't clear. Let me know if there is anything I can clarify.

[NorthEast]

My next question is whether it's possible to import the global project from a remote server, specifically Bitbucket (Git)?

You can only import the project from a local or network drive - unfortunately you can't import project files directly from source control.
(Please put in a feature request to MadCap though!)

Your global/parent project in source control will be mapped to a local or network drive, and the Flare import will point to that location.

So you need to take some care in how you use imports; e.g.:
* You need to make sure that you get the latest version of your global/parent project from source control, before you import it in the child project.
* In the child project, the project import path to the global/parent project needs to work for everyone; so you may need to use the same file/folder structure (or set up mapped drives). If it points to c:\users\sarah\etc , then the import won't work for other people.

[sarahb]

You can only import the project from a local or network drive - unfortunately you can't import project files directly from source control.
(Please put in a feature request to MadCap though!)

Your global/parent project in source control will be mapped to a local or network drive, and the Flare import will point to that location.

So you need to take some care in how you use imports; e.g.:
* You need to make sure that you get the latest version of your global/parent project from source control, before you import it in the child project.
* In the child project, the project import path to the global/parent project needs to work for everyone; so you may need to use the same file/folder structure (or set up mapped drives). If it points to c:\users\sarah\etc , then the import won't work for other people.

Oh, yikes. This seems to defeat the entire purpose. I have a team of ~20 writers, and multiple people will be working in each child project. Not to mention the technical editors that work in all of them. Being able to point a child to the parent from source control seems like the obvious (only?) solution to me. I'll definitely open a feature request, thanks!

[ChoccieMuffin]

If it's any help, I have managed to get round the problem of the absolute path in the import file.

Editing the file in a text editor (I like Notepad++ but you could easily use Flare's internal editor) you can change "C:\Users\Sarah\Globals\Globals.flprj" to "..\..\..\Globals\Globals.flprj" (I might have put in too many "..\" so do test!) and save that import file to your source control.

Because we do our source control outside of Flare, we also include all import files in our "ignore on commit" list. This does mean that we lose some of the functionality you can get with import files, but keeping our import files "clean" is much more worthwhile for us. I don't know if you are able to add files to an "ignore on commit" list in Flare's integration.

And there is always going to be some bits of a process that aren't managed automatically within whatever program you're using, so you need to ensure that the process for your team includes updating their local versions of any projects that are used in GPL before generating output. (It actually makes sense that this isn't an automatic step, as there are circumstances where you specifically don't want to use the very latest version from source control, for example if you are working on updates in the project that is imported at the same time as you're working on the project it's imported into.)

[Jeong]

Oh, yikes. This seems to defeat the entire purpose. I have a team of ~20 writers, and multiple people will be working in each child project. Not to mention the technical editors that work in all of them. Being able to point a child to the parent from source control seems like the obvious (only?) solution to me. I'll definitely open a feature request, thanks!

Just on this note, the Flare team's recommendation I think works well. Instead of putting your Flare projects somewhere in your home directory, if everybody agrees to make C:\Flare\ their Flare root, then you shouldn't have a problem.

Regarding not being able to import from source control, that's a small hiccup. When you bind a project to source control, every time you open it, by default, you'll get a prompt asking you if you want to pull the latest updates from remote. If you have reason to believe that remote and local are out of sync, then pull. Now, your C:\Flare\GlobalProject\ directory is a mirror of remote. Then, when you double-click your import file in your child project (say, C:\Flare\NewProject1), then it's going to do a "local" import from C:\Flare\GlobalProject\ to C:\Flare\NewProject1. Personally, I haven't felt this to be a problem.

The real issue is that you'll likely run into some problems using Flare's in-built source control. I'm only 2 months in, and most of my problems were related to other issues that I have been a serial complainer about, but just recently I hit upon a showstopper that prevented me from opening my project at all (thankfully, it was a child project). Without knowing the internals, I was lost. I'm in the process of unbinding the projects from source control within Flare and am doing a little bit of extra reading and note-taking to support my decision to use other git tools to for source control.

Supplementary question for all, and why I came to this thread: Any known issues / bugs / advice relating to External Resources in Flare? Since I'm using git bash, I'm trying to decide what to do with my image files... whether git lfs is preferred over External Resources. External Resources seems easier to manage, tbh, but after being burned so many times with Flare, was wondering if there was some consensus in the community about whether External Resources is a good idea or not.

[NorthEast]

Just on this note, the Flare team's recommendation I think works well. Instead of putting your Flare projects somewhere in your home directory, if everybody agrees to make C:\Flare\ their Flare root, then you shouldn't have a problem.

We handle this by getting everyone to set up a drive mapping to their local projects folder, so it doesn't matter what folder structure we use. For example, I might map my P:\ drive to c:\whatever\I\like\projects\, and the project import is set up for P:\SomeProject\SomeProject.flprj.

[Jeong]

That's a great tip. Forgot you could do that. Cheers Dave.

[techwriter31]

At some point we may be moving to Git. Does anyone know if Git submodules would be an alternative to the Flare global project linking / manual import methods?

As mentioned above, we currently use SVN externals to bypass Flare completely for importing our global files into our local/child projects, and I'm wondering if Git submodules could achieve the same thing?