Internal error: Failed to load Master TOC - #2

[Andrew Heard]

I now have the same problem as described in WebHelp Topic: Internal error: Failed to load Master TOC (Originally posted on that board but moved here as it isn't a WebHelp etc. problem)

No Toc 1.png

We have two users, two PCs, collaborating on source control.

Flare on my PC was OK, but my colleague was getting this problem when trying to open the TOC or build HTML Help. We just synched our projects and now I have the same problem. [Observation - before the issue arose, my .fltoc was not binary (could open and read in TextPad). Now the .fltoc appears to be binary (garbage in TextPad).]

I can replace the .fltoc file with one backed up on my machine about a month ago and it works just fine - plain text.

Smells of a corrupted .fltoc. Any other suggestions?

[LTinker68]

Was the TOC file ever edited outside of Flare? There was another post about someone doing that and the other program apparently saved the file as UTF8 even though the first line of the file said it was UTF16 or vice versa -- can't remember which way it's supposed to be. I think the error message was a bit different, though.

[Andrew Heard]

No, it wasn't ever edited outside Flare.

The TOC was generated on my PC, checked into source control from where it was loaded onto my colleague's PC. The TOC (and consequently the HTMLHelp build) never worked on her PC. Her version got checked into source control and thence onto my PC. Source control rollbacks don't work.

FYI support are onto it.

[trent the thief]

Hi Andrew,

I'm also experiencing this issue.

I opened up TOCs in a hex editor as my text editor also choked on those that Flare sneered at.

I was under the impression that the files were pure ASCII xml with a fltoc extensions.

There are some garbage characters at the top of the file. Changing those characters to match those in the projects I own allows me to work with them. However, once I commit the project, the owner of that project can no longer use that TOC. A TOC must be transferable.

This requires an explanation. When I commit a project, if another team member cannot fetch that project and compile it, that indicates a serious issue that could have serious repercussions. If I fall onto a subway track tonight and the other writer can't publish, I'd like to warn you ahead of time that an extremely angry product manager is going to visit La Jolla with a look to knocking it off into the water.

Bring your water wings, guy. He's mean. Hey, can you look down onto Black's Beach from there?

[Andrew Heard]

AHA!

Yes, this is serious. It has recurred for us; I am the only one who can edit the TOC and generate compiled help.

Trent - are you submitting a bug report?

[trent the thief]

AHA!

Yes, this is serious. It has recurred for us; I am the only one who can edit the TOC and generate compiled help.

Trent - are you submitting a bug report?

It's in the system now.

The problem is that from somehow, we're getting UTF-16 project imports from Frame, while Word is giving us UTF-8. Flare can read either one. Projects generated by default from Flare are UTF-8.

The kicker is that the imported files have byte-ordering prefixes and those from Flare do not.

I had to use a hex editor to open the corrupt files. I think everything, especially if you need to mix two types of projects under a master project, should be UTF-8.


This file is for project built from a word import:

UTF-8-with-byte-ordering.png

This file is a default generated for a dummy project:

UTF-8-no-byte-ordering.png

UTF-16 with byte-ordering prefix:

UTF-16-with-byte-ordering.png

I don't know why such projects the other writer created will compile on their system and not on mine. Both of us can compile the dummy project. We at first suspected it was some oddity from CVS, but we've ruled out that issue since we can both compile the dummy.

For expediency, we're going to manually normalize the files as UTF-8 without byte-ordering.

It isn't hard to do if you know a little perl and have a hex editor. But it's going to take time.

[jwatson]

Trent - you said "It isn't hard to do if you know a little perl and have a hex editor. But it's going to take time."

Why do you need perl? When I changed my TOC to UTF-8 I just edited it in Notepad, and then it opened in Flare without the error. Just curious.

Jenny.

[trent the thief]

Mine UTF-16 files were in double-byte format. That's what those columns of zeros are. After stripping the zeros with a little perl script, I discovered that a couple had broken CR/LFs (0d 0a). The files that were corrupt wouldn't open in a text editor.

[Em0606]

Thank you, thank you, thank you!

Guys, I didn't understand everything in the above post, but it was enough to figure out that if I edited each of my corrupt TOCs in Notepad to remove those little square thingies, then my TOCs would no longer be corrupt!
This has been a huge issue for us and to have a solution is fantastic!

Thanks again!

Emily

[forfear]

What source control are you using, TFS?

[trent the thief]

We're using CVS.

[JasonSTI]

We encountered this problem with source control as well. The UTF-8 files were working fine, but the UTF-16 TOC files were getting garbled. We changed the source control to check the FLTOC files in as binary instead of plain text, and this solved the issue. We never saw those junk characters at the beginning of the file, however. In Notepad, the line breaks were being replaced with three boxes, so we had to go through and replace these with line breaks again before checking them in so Flare could read them.

Thought I'd throw that out, in case anyone has the same issue.

[trent the thief]

We ended up cleaning them all. I wrote a perl script to make the change. It is odd though.

[Andrew Heard]

So is this a source control problem, a Flare problem or a local system problem?

We've just had a recurrence of same. Cleaned up (thanks for the tips, TtT) by removing leading chars, hex 00 and extra groups of line breaks in TextPad.

Question is - why only the TOC file? There are thousands of other UTF-16 files in a project.

Need to get this sorted - have submitted another support case.

____EDIT____

My Perforce guru colleague suggests that it may be because the fltoc file is not a true text or xml file, thus confusing the source control.
Have set one version to binary and another to text (8-bit). Will report back.

[trent the thief]

So is this a source control problem, a Flare problem or a local system problem?

We've just had a recurrence of same. Cleaned up (thanks for the tips, TtT) by removing leading chars, hex 00 and extra groups of line breaks in TextPad.

Question is - why only the TOC file? There are thousands of other UTF-16 files in a project.

Need to get this sorted - have submitted another support case.

____EDIT____

My Perforce guru colleague suggests that it may be because the fltoc file is not a true text or xml file, thus confusing the source control.
Have set one version to binary and another to text (8-bit). Will report back.

Hey, thanks! Great find. I never considered how source control was dealing with the file. I'd sure like to find out once and for all what's breaking those darned things.

[Andrew Heard]

This could be the final word - save the fltoc as binary in source control.

This recommended by Madcap support (and my local source control guru).

Submitted as binary - can now be shared without breaking. Can also do a diff in text mode, so I'm hoping that it will be mergeable.

Incidentally, saved a couple of other test fltocs as Text (UTF-8, I believe) - these were not shareable and returned the same error as the UFT-16 versions. When converted to binary they were fine.

[trent the thief]

Great work! Thanks, Andrew!!

This is probably a good idea for all of Flares oddball extensions.

[helen]

Just to update this - I've been in touch with MadCap support and it's now a full fledged bug. Please add your weight to this so it gets fixed soon and the UTF-16 related pain stops.

TOC files should be encoded as UTF-8 (after being edited) instead, we are encoding them as UTF-16. This is a bug and I have passed it on to our development team.

[NorthEast]

I had this problem a few years ago using VSS and found out the binary option worked (pity I didn't spot this thread earlier). Anyway, it doesn't have any knock-on effects.

[helen]

Yes the binary option would work - however it's not helpful when it comes to many popular tasks such as merge and diff. There's no reason to use UTF-16 so hopefully MadCap will fix this up for us.

[trent the thief]

Yes the binary option would work - however it's not helpful when it comes to many popular tasks such as merge and diff. There's no reason to use UTF-16 so hopefully MadCap will fix this up for us.

How often does a person need to diff and merge a TOC? I've need to do that to a few source files, but never had it come up in a TOC.

But still, I'd be great for them to fix this.

I submitted it long ago. I'm glad someone else is yelling about it

[helen]

Granted not often, but that's not an excuse. It's just another overhead for those of us using source control systems so hopefully it will be fixed soon.

[rekiwi]

I'm conflicted here: Glad to see this UTF-16 issue get some attention after so many years. Frustrated to see folks still suffering from it after so many years.

Once I figured out what was happening, I've always been puzzled that Flare declares its XML files as UTF-8 ("encoding=utf-8") but then saves them with the UTF-16 byte order mark (BOM), causing much wailing and gnashing of teeth when using revision control systems like CVS that are bad at handling binary files and dependent on file endings.

[Andrew Heard]

How often does a person need to diff and merge a TOC?

When more than one writer is editing the TOC. We now have three writers working on the same project simultaneously, all of whom could have reason to modify the TOC.

Yes the binary option would work - however it's not helpful when it comes to many popular tasks such as merge and diff.

Perforce will do a diff and merge on a .fltoc saved as binary.

UPDATE

Perforce will do a diff and merge on a .fltoc saved as binary.

Merge corrupts the file

[Andrew Heard]

Update -

Problem still there - when TOC is merged, it gets corrupted and cannot be shared. Even when set to binary, sets to UTF-16 and adds junk characters.

In this case, it also overwrote a large portion of the file with junk, so that the original content is unrecoverable.

TOCJunk_text.png

TOCJunk_hex.png