MOTW and large text files

scooter262 · Post by **scooter262** » Thu Feb 04, 2010 10:35 am

I've got an interesting situation.

I'm generating a Help system (WebHelp). Previously, I had turned off MOTW, because I needed to link to text files. These files can be
quite large (some are close to a gigabyte). Now, I have turned on MOTW, and converted these text files to HTML, by wrapping them in simple
HTML tags.

The problem is that now, whenever Flare deals with these files, it does some analyzing/processing, which takes a long time for the large files. Even
to navigate to the subfolders that contain these files takes a while. At first, when I was generating the project, I thought Flare was hung.
However, I was finally able to create the target, but it took over 25 minutes, when the prior time was under 3 minutes.

I'd like a way to be able to tell Flare to treat these large files as it did the text files -- sort of as black boxes that don't need any
analyzing or processing. Is that possible? If not, is that something MadCap could investigate as an option for future releases?

Right now, my solution is to keep dummy files in the project area, and then copy the real files into the output after I generate the target. But
this a hack, requires a manual step, and therefore is more error prone than being able to cleanly generate the output.

Post by **LTinker68** » Thu Feb 04, 2010 10:51 am

Are the files stored in your project or outside the project?

scooter262 · Post by **scooter262** » Thu Feb 04, 2010 10:56 am

I now store the HTM files outside of the project. I keep dummy files in the project, so that I don't get errors when generating. Here is an example of the dummy files:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=windows-1252">
<META content="MSHTML 6.00.2900.3527" name=GENERATOR></HEAD>
<BODY><PRE>
blank - template for HTML wrapping
</PRE></BODY></HTML>

For example, this file might be called "details_2009_06.htm" and outside the project, I have a file with the same name that contains lots of text. Previously, when these were text files ("details_2009_06.txt), I kept them inside the project.

Post by **LTinker68** » Thu Feb 04, 2010 11:11 am

Hmmm. I was thinking if they were the project then Flare is running Analyzer on the files, and since they're so large, it takes awhile to do it, especially since they're not HTML topic files. I wonder if there's a way to tell Analyzer to ignore a folder and not scan it? If not, then I wonder if temporarily adding a conditional tag to that folder that's excluded from the target will keep Analyzer from scanning the folder. If so, then you could apply the conditional tag to the folder while writing then remove it just before building.

I've never actually studied how Analyzer works, though, to see if it will ignore folders that aren't included in the "primary" target. Which means the targeted output might also have to be your primary output. Anyway, it's something to try in the meantime. Otherwise, adding the real files after generating the output might be your only recourse, in the short term. Certainly submit a feature request at http://www.madcapsoftware.com/bugs/submit.aspx asking for the ability to exclude files/folders from an Analyzer scan.

Madcap Guru · Post by **Madcap Guru** » Tue Feb 23, 2010 8:27 am

Why are you using MOTW? Is the End result of the files going to be on the users local system? If not, I would Just turn MOTW off

scooter262 · Post by **scooter262** » Thu Mar 04, 2010 8:54 am

We need MOTW on to avoid a security warning that users get the first time they access a page with MOTW off, if they use Internet Explorer.

Post by **LTinker68** » Thu Mar 04, 2010 9:38 am

So your users are running the help from their local computers, not from a location on a public web server you're hosting?

scooter262 · Post by **scooter262** » Thu Mar 04, 2010 10:09 am

The application and help are on an appliance at each customer site. Their users access the help from local computers on their company's LAN. In testing, we have definitely seen the warning pop up, typically on first usage.

Post by **LTinker68** » Thu Mar 04, 2010 11:27 am

Just out of curiosity, did you ever open those HTML files from within Flare? You said you converted the files to HTML files by wrapping them in html tags. If you're going to leave them as HTML files, then you might want to open them in Flare. Flare will then "convert" them to XHTML files and will add some info at the top of those files like they do other topic files. Doing so might make Analyzer review them faster. If they were strictly text files with a lot of text, then it shouldn't take long to scan them, but if the tags in them aren't properly formed/terminated per HTML rules, then it might slow Analyzer down because it has to do more work to evaluate the structure of the file.

It could just be, though, that it's the size of the files that slows Analyzer down. And if your project isn't on your local computer, then speed of the network could also be slowing it down. If doing the above doesn't help, then submit a feature request to MadCap at http://www.madcapsoftware.com/bugs/submit.aspx asking for the ability to mark a file or folder as exempt from scanning.

scooter262 · Post by **scooter262** » Thu Mar 04, 2010 12:58 pm

Here's what all the files look like:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=windows-1252">
<META content="MSHTML 6.00.2900.3527" name=GENERATOR></HEAD>
<BODY><PRE>
:
Tons of text...
:
</PRE></BODY></HTML>

If I was to let Flare work on them, I would remove the first two lines, which I hard-code into the files so the links to them will work with MOTW turned on for the project.

I do run the project on my local drive. However, either Flare is so slow, or the files are so large, that if I'm in the Content Explorer pane of the project in Flare, even clicking on a containing folder that has a bunch of these files takes a very long time before it even lists the files.

Maybe I should test converting one or two of the files to XHTML, but then even if that worked okay, I would need to convert all 150+ files to see whether the sheer size of the data would still keep things inordinately slow.

Post by **LTinker68** » Thu Mar 04, 2010 1:04 pm

scooter262 wrote:Maybe I should test converting one or two of the files to XHTML, but then even if that worked okay, I would need to convert all 150+ files to see whether the sheer size of the data would still keep things inordinately slow.

Or convert a couple of them and leave placeholders for the rest. If there's no appreciable difference with just a couple of files, then convert a handful more to the project, see how it handles them, and so on. You might also want to remove the first few lines of code and let Flare insert what it needs to, just to make sure there isn't any conflict between the original code and the Flare-inserted code. I think the MOTW line, for example, isn't actually added to topics until you generate, so it might be better to let Flare insert that at build time instead of having it in the authoring environment.

Post by **NorthEast** » Fri Mar 05, 2010 2:55 am

With regards to the speed of the build; perhaps one factor could be be that Flare is scanning these huge topics when it's building the search database.

So, does the build speed improve if you omit these topics from the search database?

To do this, open the topic's properties, select the Topic Properties tab, then remove the tick from Include topic when full-text search database is generated.

I don't know if this will make a big difference, but it's worth a try.

MadCap Software Forums

MOTW and large text files

MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files

Re: MOTW and large text files