Flare, store Knowledge Base articles in an SQL Database

[deucalion0]

Hello all!

I have been researching the capabilities of Madcap Flare as I am interested in using Madcap Flare for a Knowledge Base.

I have started my trial and I have installed and been playing with the installation.

I have followed the getting started guide and watched the YouTube videos and so far it is relatively straight forward to use.

My main concern is that I would like to store the topics in a database so that they can be queried against and I am not sure if this is possible.

Digging around I see that the data seems to be stored in a flat file.

Can anyone shed some light on the possibilities of using an SQL database to store KB articles instead of the flat file approach?

Any advice is much appreciated!

Thanks!

[RamonS]

While SQL has benefits when it comes to querying, there are plenty of full text search engines that accomplish about the same with flat files. Have you tried Flare's built-in search capabilities? May not be exactly what you had in mind, but typically the need is to query text for keywords. What you could do is store the URL of a topic in SQL and add as much meta data to the record in the database as you need. I'm not entirely clear what you want to query the topics for. Out of the box there is no means to store topics in a database. Main reason is that the topics are to be served up using a web server, having files in SQL Server adds incredible amounts of overhead and likely would prevent using JavaScript based features.

[deucalion0]

While SQL has benefits when it comes to querying, there are plenty of full text search engines that accomplish about the same with flat files. Have you tried Flare's built-in search capabilities? May not be exactly what you had in mind, but typically the need is to query text for keywords. What you could do is store the URL of a topic in SQL and add as much meta data to the record in the database as you need. I'm not entirely clear what you want to query the topics for. Out of the box there is no means to store topics in a database. Main reason is that the topics are to be served up using a web server, having files in SQL Server adds incredible amounts of overhead and likely would prevent using JavaScript based features.

Hi RamonS, thank you very much for your answer, it is much appreciated.

I have tested the search capability and it seems to work very well and allows the ability to AND and OR quite well.

I will explain a little more about what it is I am trying to achieve.

I may have a web page that need to pull certain topics or certain topic fields to that page.

I will have to think about how to tag topics in a way I can collect the topics/fields that are relative for my page.

I have never seen this method of storing data before, this flat file form.

Can you tell me exactly the name of the file where the data is stored? I added a specific string to a topic and then searched the files for that string and found these three files:

SearchPhrase_Chunk0.js
SearchStem_Chunk0.js
SearchTopic_Chunk0.js

Is one of these the flat file data I need to work with?

I have analysed each of these files and some of the data makes sense a lot does not.

I agree that my initial idea is probably not the best due to the overheads you mention. I am very interested in pursuing an idea similar to your suggestion.

Thanks for your help!

[RamonS]

As far as Flare output is concerned, it is essentially nothing else than plain ol' web content. A bunch of HTML files that are made a bit smarter with JavaScript and a bit prettier with CSS. All that a web server can handle on its own are flat files. I'm a bit surprised that you are surprised. Without server side code there is no means to make a connection to an SQL database. That, of course, can still be done using Flare when you add server side code to your pages, such as PHP.
I'm no expert in how the Flare search is constructed, but I recall that I read about it in Thomas Tregner's book "MadCap Flare for Programmers". I have the copy at work, so I cannot exactly confirm what about search is covered, but I recall that it went into great detail what all the file contents mean. In any case, it is a very good book if your are interested in lifting up the hood of Flare.

What you are after is to find content in Flare that matches some sort of query. Do I further understand correctly that you then want to retrieve that content and display it in some form? Ideally, you want to make the chunks of information individual topics. That saves you from taking parts of a topic and somehow processing those into a new web page that a browser can display. To do that in the confinements of Flare seems rather difficult. I case you come across snippets, those only exist within Flare and are not available as individual items after compile.

My advice is to look into server side code such as PHP or the like that connects to a database that can run more complex queries across tags and other meta data stored in a table together with the URL of the topic. If the goal is to display your own dynamically generated page from content (text, images, media) I'd take Flare out of the equation entirely. Craft plain or preformatted text that you store in a content table. Create tags and meta daa that link to one or more content records. Based on the query submitted pull the content needed. If you use MSSQL you may want to engage the built-in full text search which works quite well, although in my experience it is rather dumb when it comes to handling of stop words and ranking. Once you have the content pieces arranged process them into HTML and return to the client.

What I'd do is add server side scripts to Flare output and provide on each page a means to conduct a more complex search using any search pattern you can make work. The result is returned as links to topics within the Flare output. That does not preclude using the JavaScript search files that Flare generates, but it all depends on how helpful the content of those files is for you.
Before that conduct a gap analysis of what you need and what the Flare search does not provide. There may not be much that is amiss and then the question is if it is worth the effort to fill that gap with custom code and a database backend. The only gap I could see is allowing for extensive use of alias terms or for searching of meta data contained in jpeg image headers. That may be easier than stuffing tags into the topics that are content for the Flare search but get hidden at display time in the client.

[deucalion0]

As far as Flare output is concerned, it is essentially nothing else than plain ol' web content. A bunch of HTML files that are made a bit smarter with JavaScript and a bit prettier with CSS. All that a web server can handle on its own are flat files. I'm a bit surprised that you are surprised. Without server side code there is no means to make a connection to an SQL database. That, of course, can still be done using Flare when you add server side code to your pages, such as PHP.
I'm no expert in how the Flare search is constructed, but I recall that I read about it in Thomas Tregner's book "MadCap Flare for Programmers". I have the copy at work, so I cannot exactly confirm what about search is covered, but I recall that it went into great detail what all the file contents mean. In any case, it is a very good book if your are interested in lifting up the hood of Flare.

What you are after is to find content in Flare that matches some sort of query. Do I further understand correctly that you then want to retrieve that content and display it in some form? Ideally, you want to make the chunks of information individual topics. That saves you from taking parts of a topic and somehow processing those into a new web page that a browser can display. To do that in the confinements of Flare seems rather difficult. I case you come across snippets, those only exist within Flare and are not available as individual items after compile.

My advice is to look into server side code such as PHP or the like that connects to a database that can run more complex queries across tags and other meta data stored in a table together with the URL of the topic. If the goal is to display your own dynamically generated page from content (text, images, media) I'd take Flare out of the equation entirely. Craft plain or preformatted text that you store in a content table. Create tags and meta daa that link to one or more content records. Based on the query submitted pull the content needed. If you use MSSQL you may want to engage the built-in full text search which works quite well, although in my experience it is rather dumb when it comes to handling of stop words and ranking. Once you have the content pieces arranged process them into HTML and return to the client.

What I'd do is add server side scripts to Flare output and provide on each page a means to conduct a more complex search using any search pattern you can make work. The result is returned as links to topics within the Flare output. That does not preclude using the JavaScript search files that Flare generates, but it all depends on how helpful the content of those files is for you.
Before that conduct a gap analysis of what you need and what the Flare search does not provide. There may not be much that is amiss and then the question is if it is worth the effort to fill that gap with custom code and a database backend. The only gap I could see is allowing for extensive use of alias terms or for searching of meta data contained in jpeg image headers. That may be easier than stuffing tags into the topics that are content for the Flare search but get hidden at display time in the client.

Hi RamoneS, thank you for your detailed response.

I had just never seen this method of storing data, I am still not sure what exactly the Chunk files are, I am now reading as much documentation as possible in order to understand as much about this as possible.

I have had a look at the book you mentioned I believe it may be very useful to get all/most the information I require.

It may be a case that Flare is not the ideal solution for me, I will have to continue researching to get a better understanding about the possible solutions to meet my requirements.

Your understanding is correct. I want to take take specific Topics, perhaps those tagged as "URGENT" and display those on a page called "Urgent Topics" in descending order with the latest Topic at the top followed by the next 9 topics, some pagination with 10 topics per page. I want these Topics to also be searchable. I would perhaps repeat this process for other pages using different tags e.g. "NEWS" and "SPECIAL".

Until I understand the data used to store Topics I will not know if what I want is achievable.

You have provided some good advice on where to start!

Thank you very much for your time on this!

[RamonS]

You can still achieve that with Flare, but you either will need to create these lists manually, such as through a section in the ToC,employ server side scripting, or use hidden text that gets picked up by the Flare search. Using hidden text will not give you much control over the ranking unless you game the system and add the text "tags" multiple times to a topic.
Storing everything in a database and rolling your own site that still has a Flare-ish feel to it sounds like a lot of work. Have a look through the forums to find others who used Flare for knowledge bases.

[GregStenhouse]

It may be best to look at a plugin within the Flare editor that saves/copies/synchronizes the XHTML from a topic to your SQL database and/or back again? Or some kind of integration between source control supported in Flare like GIT or SVN and the database. That way the Flare output is separate and continues as flat files and you don't need to tinker with the Flare JavaScript or search logic, and the topics are in a database to query for whatever other purposes you need as well. Just a thought,

Cheers
Greg