How to make Flare compile faster

[wijnand]

I have had a ton of questions from people in Holland who wanted to know how to speed up flare. Some people even compare it to how Robohelp can compile faster then Flare so I thought I should share you some information:

What does flare do during a compilation?

Every single topic that you have in your project gets examined by flare, things it starts looking for are Index keywords, conditions, Conceptual words etc. As you know Flare is a very modular HAT compared to Robohelp, instead of having things centralized you can basically rip a entire flare project apart and move individual topics around without losing its index and conditions (aslong as the condition name exists in your new project otherwise analyser will give a warning).

Now imagine having a project with 5000 topics, this means that flare has to read each topic and process it before it can continue with making the HHP/HHK/ file. Now why has Madcap decided for this approach? There are a couple of probable reasons:

1. Its more reliable
2. You aren't forced to have a centralized project file that contains all the conditional information etc.
3. it becomes interchangable you can easily move project files around without having to do a lot of manual editing (unless it involves links to topics you didnt include along).

Now how do we speed these things up? There are a few things flare really looks at for a good performance (listed in importance):

Hard Disk
Ram
CPU

1. Hard Disk

The hard disk is the most important thing you have to look at when you are working with flare, a lot of people I talk to use flare on a Laptop, laptops are machines that are always trying to optimize its portability and also how long it can last with one charge. The downfall is that most Laptop components are a lot slower then a comparable desktop PC eventhough the MHZ might be the same that does not mean that it will be equally fast (unless its a workstation laptop). Most laptops have a 5400 RPM hard disk meaning it can make 5400 rounds per minute, a desktop PC has 7200 Rounds Per Minute meaning its 33% faster alone! this ofcourse does not account for the fact that there might be different seek times, spin times etc.

The biggest advise I can give for a company that has a 4000-10,000 topics is to use a really good hard disk, either use a Raid setup or get a 10,000 RPM hard disk. Raid is basically something where you buy a few hard disks that are the same size and brand (for best results) My Raid setup at home is basically 3 x 80 gig hard disks 7200 RPM setup in Striping mode this will result in 3 hard disks working together as one hard disk. Meaning you suddenly have 3 heads reading off a cylinder then just one. The speed increase in my case was that instead of waiting 20 minutes I was waiting 6 minutes.. However I have to wanr you that raid has one giant weakness. If one hard disk fails you will lose your data on ALL drives. So be carefull with it, and make backups. There are ways to solve it on a Raid level by doing Raid 5 but I think your tech support/system administrator can help you beter then I can with one post

For people interested in knowing how to setup raid:

http://www.pctechguide.com/tutorials/RAID.htm

In the case where you cannot get a RAID setup your best option is to either try to get a 10,000 RPM hard disk or try to keep your Hard disk defragged (organized). Or you can try to increase your Ram:

2. Increasing ram

Ram is used by Windows to put information that it needs to process so whenever you open a program memory is allocated to it so it can store its data in this temporary storage. Windows itself also uses Ram aswell so as you can guess if you only have one gigabytes of RAM you have a possibility that it will run full with information. Now in the old days Windows would crash on you, or it would state it ran out of memory and close down on you. Since Windows 95 there has been something called Virtual Memory, Virtual Memory basically means that Windows uses your hard disk as a temporary place to store memory from a program. The downside to this is that it will slow down your program, but it will also require Hard Disk data meaning that you will slow down your hard disk.

If you turn off your virtual memory and start up a lot of programs Windows will suddenly give you a warning that you ran out of memory and then crash on you. So how can we ensure that our hard disk is left alone by Windows XP/Vista? The simplest solution is to upgrade your Ram, On my Vista PC I have 8 gigs of ram (64 bit IS required for this) and I have turned off my Virtual Memory meaning that whenever i start up a program its completly stored into my RAM instead of it occasionally slipping into the hard disk. But you can keep this turned on cause it MIGHT crash your Pc if you suddenly run out of memory. Although I have to add Windows Vista has a lot beter memory management then Windows XP does. I have noticed an increase in performance when doing this. But again it depends completly on your situation.

For people interested in knowing what virtual memory is:
http://en.wikipedia.org/wiki/Virtual_memory

3. CPU

The only time a CPU will help you with improving the performance of Flare is when you are using a 1.2 GHZ PC. Flare is programmed in .NET software meaning that it will be performance wise a little slow in the beginning but it should speed up after a few times. CPU speed will not magically increase your compilation by 200% if you upgrade from 1 GHZ to 3GHZ although there will be a increase, it will not be by the leaps you will be hoping for. The biggest reason for this is ofcourse simple: your reading data from your hard disk, your CPU processes it then and returns it. Your hard disk will be the one carrying the most burden not your CPU. Infact your CPU will be waiting for new data as you are compiling because the hard disk cannot keep up. So only upgrade your CPU if your Flare performance seems to be a bit slughish when your actually working in your project.

Software optimisations:

AGAIN i have to stress I am not at fault here if your Pc becomes a smoking derick so if you do not feel comfortable doing this then don't do it!

1. Turn off Windows Indexing:

Windows XP and Vista especially have something called Indexing where basically windows keeps tabs on every file you have available on your computer so when you do a file search it can give you results immediatly without having to search your hard disk. The downside however is, specially with Vista is that sometimes it will be doing a search while your compiling causing a bottleneck. although I have heard that Windows DEsktop search 4.0 has largely mitigated the performance issues people have had with the indexing service of windows. If you want to see how much this affects your PC here are two guides that helps you to disable this:

XP
http://lifehacker.com/software/optimiza ... 031440.php

Vista
http://4sysops.com/archives/how-to-disa ... ws-search/

2. Turn off your Antivirus before compiling

Again Most security programs specially in the bigger companies have a very restrictive policy of basically searching your pc constantly for viruses. This requires your harddisk to be used and it will degrade your performance as you go. So before you compile try to disable the background service until your done. (this can be done within the Antivirus itself) and turn it back on afterwards but don't worry Windows XP and Vista both will complain its turned off every 5 seconds!.

3. Defrag your hard disk

Defrag your hard disk once a week, if your files are all nicely organized it will require less disk spins to get the files together to read. If you never defrag some files might be scattered on 200 different places on your hard disk instead of just on one location meaning it has to make 200 rounds to find all the files.

windows XP and vist ahve disk defragmenters so use these on a weekly basis, I tend to run it friday night before i shut off my PC.

For people interested in fragmentation:
http://en.wikipedia.org/wiki/Disk_fragmentation

4. Active Directory

I was once working at a Bank (no names!) and every 6 minutes my Pc would have a checkup with its active directory to see if I was keeping all my security settings in place. The results where basically that my entire PC would just not respond while it was doing this check thus slowing me down tremendously. so if this is your scenario ask your admin if there is a PC where you can do this without the slow downs. But usually your stuck with this sad situation I think i wasted 2 entire days just waiting for my PC to be unfrozen again!

Well this is my first start for now I hope its usefull, I will be adding more information when neccesary but i think this is a good start!.

Wijnand

[RamonS]

Thanks for posting this. There are other benefits from the modular approach. Two things come to mind right away:
- master pages
- proxies

Those get built and inserted during compile for each topic, which means that besides finding all the other things already mentioned an entirely new topic file for output is generated.
Also, when building CHM output, Flare first generates a help project that then gets compiled by the proprietary Microsoft HTML Workshop compiler.

In regards to defragging a disk, don't use the Windows defragment tools. They stink and do only half the job. Use a tool like JKDefrag (http://www.kessels.com/Jkdefrag/), which, of course, is made by a Dutch developer.

[wijnand]

I rather not post or make commercials on defrag tools, but I use Diskkeeper

http://www.diskeeper.com/defrag.asp

[Kleb]

Yes, JKDefrag is a great program. Not only is its default defrag mode better than the Windows defragger, but it has several other command line modes that can significantly speed up your computer.

The mode I use is

Code: Select all

jkdefrag -a 8

This will rearrange your files by size. How does this speed up your drive? Well, the longest part of reading a small file (.htm and other xml-like files) is getting there (moving the hard drive read heads to that location).

By placing similarly sized files together means that all your .htm files will be very close together meaning much shorter seek times. The same thing happens for Windows drivers, DLLs, and exe's meaning that your Windows boot up time will be quicker.

My computer is fast, but a lot of things need to load at startup, which means that each program is asking the HDD to load data at the same time. If the data is closer together, it will take much less time to read it all. My bootup times were on the order of 10 minutes (to where you can start a program and not wait a minute for it to load). Just doing jkdefrag -a 8 dropped the boot up time to 5 minutes!

One note: it will take a long time to rearrange all the data so it's best to start this mode and leave it running overnight. Also, it's an all-or-nothing reorganize, so if you interrupt it, it will have to start over again.

Lastly, this complete reorganization does not need to be run that often. Once every month should be plenty. For defrag maintenance in between, running jkdefrag normally (or better yet, to not disturb the reorganization as much, "jkdefrag -a 2") is fine (weekly). Also, the reorganization will cause fragmented files. This is on purpose to get as much data in the fastest area of the HDD.

[forfear]

This is a really good forum post!
i am compiling a master project with the sub projects linked to source control, and building a PDF (240 pages) takes 5 minutes or so each time i click build target.
The recommendations are great and I'll give this a shot.

[Andrew]

I think another thing to mention here is that there are diminishing returns with any of these things if you are only addressing one issue. If you already have a good, fast disk (or array), but not a fast processor, you will get more out of a faster processor than you will out of even faster disks.

I ran perfmon while compiling my primary project (HTML Help, 6300 topics, a master page has been applied to all the topics, and the project uses a couple of dozen conditions -- also note, I am not running an on-access virus scanner). I set it to count 4 things: disk writes, disk reads, queue length, and processor time (as a percent of total available). See the linked screen shots for the whole data capture.

Over the course of the entire compile (11 minutes), here are the average values (NOTE: this is one type of compile with a single project with a particular set of data; it may or may not be representative of other compile types and data sets -- if you are curious, you can always run perfmon on your own compile using these counters):
Disk Writes: 740 KB/sec
Disk Reads: 560 KB/sec
Disk Queue Length: 1.32 requests outstanding
% Processor Time: 39% (out of 50% total -- this is a dual-core machine, but Flare is basically single-threaded, so it can only run on one core at once)

The two that interest me most are the queue length and processor time (queue length, by the way, is the number of i/o requests that are currently waiting for the disk to finish what it doing before they can be read or written). Clearly, there is some waiting for the disk going on (queue length of more than 1 means there is waiting), but there is just as clearly some fairly heavy processor use. Going beyond the averages, if we look at the charts, we can see a fairly steady pattern of 1 or 0 queue length, with periodic spikes up as high as 20 requests or so. A faster disk won't do much to help those queue depths of 1 or 0, but it will sure help out those spikes. A faster processor won't speed up the areas where usage is less than saturated, but it sure will in areas where it's pegged. In fact, a faster processor might create higher queue depths (because it can serve them faster), whereas a faster disk might increase processor time numbers, because the CPU has more to chew on from the disk. It's a balancing act, and the idea is to find the ideal balance within your budget for hardware.

I'm not really sure if that helps anyone; I'm very curious about measuring the performance of Flare, though, and if anyone has some better ideas for counters etc., I'd love to discuss it.

[Edit: I ran this compile with Windows Search Indexing turned on, on Windows Vista. So even with search indexing on, queue lengths were fairly low -- possibly due to the prioritized i/o added in Vista.]

Disk Queue:


% Processor Time:


Disk Reads (Bytes/Sec):


Disk Writes (Bytes/Sec):

[Andrew]

As for the advice on defragmenting: fragmentation isn't all that likely to cause many problems with a Flare compile, because most of the files are too small to fragment significantly (unless your fragmentation is ridiculously large), and generally, fragmentation mostly hurts large sequential transfers (which compiling is most definitely not doing). Generally speaking, both how large and how full your drive is matters a lot more than how fragmented it is: the more empty the drive is, generally the better your performance is. Still, defragging doesn't hurt, and may help a tiny bit. Just make sure you don't spend more time defragging than you gain in compile performance.

The by-file-size defrag option is an interesting idea, since it could theoretically keep all of your project files close together -- assuming they are of approximately the same size, of course, and assuming you do not have lots and lots of other files of a similar size (whether from different projects, or from some other program). Having all the files in a small disk area significantly reduces seek time for the drive, which will help perf a bunch, I'd wager. I may give that a shot sometime and run some before/after perfmon counters. The one question I have about that is will it interfere with the i/o optimization that Windows does automatically (specifically with Vista, and to a lesser extent, XP)?

[RamonS]

Would be nice to see the same test run on XPSP3 using the same hardware. From my (unscientific - I used the system clock to measure total time only) tests Vista is about 1/3rd slower even with SP1 applied.
In regards to the defragging, you are right about the small files that Flare uses, but as stated earlier, using for example JKDefrag you can place small files on the faster sections of the hard disk. And besides that, defragging helps overall system performance, especially when using badly designed file systems like NTFS (for example placing the allocation table on the first track rather than in the middle of the drive, which would decrease seek times and also put the data on the most reliable area of a disk).

[i-tietz]

That's all very good advice. And maybe it helps sometimes or even most of the time.
But: What do you do, if it doesn't help?
The situation is project A being compiled:
- on computer X with Flare 3.1 - 18 minutes
- on computer X with Flare 4 - 14 minutes
- on computer Y with Flare 3.1 - 7 to 8 minutes
- on computer Y with Flare 4 - 26 to 30 minutes
How can you explain that sort of result? Who has the better hardware? Who has the better defrag tool? Do they count for Flare 4 only?

There must be more to it than meets the eye ...

BTW: The project has a masterpage for the glossary only and absolutely no proxies and dropdowns - it's an old RH project so it contains none of those possibly "obstructing" features.

[RamonS]

Wow, that is interesting. Same project, same software, same runtime....do the systems run the same OS? Regardless, in one case 3.1 is noticeably faster and in the other case it is 4.0. I think that throws any 3.1 vs. 4.0 statements out the window.

[i-tietz]

That's the facts of my case - I reported it a few weeks ago. Gives you a hint on how complex that problem is and how long it might take to solve it ...
I am working for a software company myself and performance problems are usually most hard to solve.

[wijnand]

It honestly sounds like there is something on the other computer thats bogging it down.

Usually when a computer shows compile times like this I tend to think the following:

1. A process is running on your PC thats demanding I/O
2. The computer has a dying hard disk

Hard disks that are slowly dying basically become very slow and unresponsive. A good tool to use is Speedfan which has a S.M.A.R.T monitor this will tell you how the health of your hard disk is how many errors its causing and if the performance is up to par. I use this tool on most my machines because so far it has given me very good warnings when a hard disk was about to croak. If your hard disk isnt the problem another thing you can do is check how many processes each computer run and see if they are identical. If they arent I am going to lean against software. A poluted registry is also a good reason why a computer might slow down horribly because its having problems finding files and thus starts bickering over them . If you dont use Vista i recommend Regcleaner.


Windows Search 4.0 (which is being added in Vista SP2) apparantly addresses a lot of the problems people had with the windows Indexing service that was included in the RMT . So you will notice in that respect it will be a lot faster.

[i-tietz]

It honestly sounds like there is something on the other computer thats bogging it down.

nope - we can reproduce it on all of our 6 PCs. The other PC was in MadCap support.

Apart from that: Flare 3.1 is on the same harddisk and on the same system ... how comes that with Flare 3.1 compiling is faster with our PCs than with those at MadCaps support department?

Just face it: It's probably not a hardware problem ...

[wijnand]

if you are reproducting it then the next question is what are you doing to make it slow down so badly on one system?

[KevinDAmery]

At that point, I would suspect two things:

1) You probably have faster systems than Madcap does (which is why 3.1 is faster for you)

2) There is probably something installed in your systems that Madcap doesn't have, and that something is getting in the way somehow. I'm not sure what it might be - possibilities include the version of .NET, the version of Windows, virus / security software, service packs, a dll or other system resource getting updated / over-written by another piece of software.... Not a fun list to track down, sadly.

[wijnand]

You could try hijackthis and check what processes are different on the machines

http://www.trendsecure.com/portal/en-US ... ckthis/qsg

[i-tietz]

reading my last post agian I think it might be misunderstood.
In my post in which I gave the runtimes on two different computers, computer X is the MadCap one and Y is mine (and those of my 5 colleagues).

[wijnand]

what are your system specs?
Are you in a Active Directory?

Its hard to work with just the results if you do not know the things that are resulting to this output

[i-tietz]

what are your system specs?
Are you in a Active Directory?

Its hard to work with just the results if you do not know the things that are resulting to this output

Are you developer? That sort of enthusiasm for solving a riddle ...

I communicated all those pieces of information to the MadCap Support weeks ago. Undoubtedly they will take care of it. And if they need any more of info, I will tell them. But I have given that problem a rest, since I won't run into it seriously until almost March 2009. That leaves the Madcappers with ample time.

[wijnand]

yes I am a developer, I program in C# / C++ / Java

And I like solving problems!

[trent the thief]

Hi,

Use the Deferred Procedure Call Latency (DPC) checker from http://www.thesycon.de/eng/free_download.shtml and the sysinternals procexp utilities to verify that there are no driver-related problems on the slow system.

I've got a nice fast Dell that falls to its knees every 45 minutes or so because the only available driver for the DVD R is hosed. Once I disable that drive my box runs fine.

It's bloody annoying, that, but I'll live with it. I only use the drive to archive projects.

[i-tietz]

Use the Deferred Procedure Call Latency (DPC) checker from http://www.thesycon.de/eng/free_download.shtml and the sysinternals procexp utilities to verify that there are no driver-related problems on the slow system.

mmmhh ... my "system" is slow only with Flare 4 (now 4.12). And at compiling with Flare 3.1 my system is even faster than the PC at MadCap Support.
How could that be a driver problem? I mean, ... the drivers are always the same, aren't they?

The only thing that's different is the Flare version ... so that's where the reason must be sought after ... and I believe, that's what MadCap is doing (posted a bug report). Our support contracts last till October this year - maybe we will get Flare 5 that way and the problem is solved.

[trent the thief]

Beats me. I throw up my hands with the .NET stuff. I figure anything that might help diagnose the problem can't hurt.

How many version levels of .NET are installed?

[i-tietz]

How many version levels of .NET are installed?

If you tell me what that is, I might be able to look it up.

[trent the thief]

Look in your Add/Remove Programs for Microsoft .NET.

It's like the past where you needed the VB or VC runtimes, except it doesn't work as well. Sometimes you run into situations where you need more than one version to get the best performance out of an application.