WebHelp and Search Engine Optimization

[nfberg]

Being our Help is publicly available and an integral part of our site we are looking at methods to make WebHelp more crawl-able. Since all the content of help is buried under javascript and can't be directly crawled by the search engines we are looking for suggestions on how get past this.

Thanks

[jbrock]

nfberg, Did you ever find a way to do this?

[LTinker68]

Technically speaking, I don't think it's the Javascript that's causing problems for the search engines. It's most likely because the whole help is in a frameset. I thought most modern search engines were okay with framesets now, but maybe not. If not, perhaps WebHelp Plus with appropriate settings on the web server might take care of that. I know if I do a Google search on Flare 6.1 bug it will return something from MadCap's knowledgebase, which is a WebHelp Plus output in a frameset. (Although when I click on the search results I don't get the frameset, just the topic.)

[jbrock]

Search engine optimization for webhelp may be one of the most important topics in this forum, I think. Most customers are going to start looking for help on Google, not in the search panel in Flare. You can have the best help on the planet, but you’ll only frustrate customers if they can’t find it on Google.

I think Flare is best-of-breed for creating help documentation. It is certainly making my team much more efficient. But the framed output creates a usability problem for customers.

Google finds pages by finding links to those pages. If it never sees a link, it won’t see the page. Therefore, you can get an idea of how well Google and other search engines index the framed webhelp by generating an xml sitemap for the webhelp site. [Obviously, Google's algorithm is more complex than an xml sitemap generator's, but the latter does give you a good high-level view of internal linking.]

For example, we generated an xml sitemap starting from http://kb.madcapsoftware.com/, the knowledgebase on madcapsoftware.com. Only 10 pages were crawled!

The xml sitemap finds so few pages for the knowledgebase because all of the links are to the frame.

• http://kb.madcapsoftware.com/
• http://kb.madcapsoftware.com/Skin/Toolbar.htm
• http://kb.madcapsoftware.com/Skin/Navigation.htm
• http://kb.madcapsoftware.com/Skin/Blank.htm
• http://kb.madcapsoftware.com/Skin/Toc.htm
• http://kb.madcapsoftware.com/Skin/Index.htm
• http://kb.madcapsoftware.com/Skin/Search.htm
• http://kb.madcapsoftware.com/Content/Glossary.htm
• http://kb.madcapsoftware.com/Skin/Favorites.htm
• http://kb.madcapsoftware.com/Skin/BrowseSequences.htm

It would be better for Google to hit the first page pulled into the frame - http://kb.madcapsoftware.com/Content/Welcome.htm - and follow the links from there. Unfortunately, the welcome does not link any deeper into the webhelp.

This is a bit of a usability issue. Madcap Software’s own documentation is probably the most authoritative. Their writers spend the effort creating and updating – and they do a great job. But other sites are going to rank higher on Google because the other sites use WordPress, some other CMS or hand-coded HTML instead of frames. In fact, this very user forum will rank higher because phpbb doesn’t use frames.

Tom Johnson wrote an article entitled “Search Engine Optimizing Your Help Content for Google” in which he tested the search results for a Flare topic. He searched for “insert concept keywords flare” in Google. Flare's webhelp article - Inserting Concepts into Topics - didn't show up. His conclusion:

As the importance of visibility on Google grows, and as companies recognize and treat their help content as an SEO asset for the online visibility and ranking (not to mention marketing) of their products, shouldn’t we put our help content on web-friendly platforms that will maximize their visibility in Google’s search engine results? Are traditional help authoring tools holding us back from realizing the SEO power of our help content?

Although Tom Johnson emphasizes the marketing benefits, I think an argument from usability in favor of moving away from the traditional frame output could be just as strong.

There's a way to make Flare search engine optimized, but I think I'll do that in a separate post.

[mrentropy]

This is exactly the topic I was searching for. Justin, where will you be posting your other article?

[jbrock]

Mr. Entropy,
I'll post something on the forums here once we're finished working on it. Here's a hint about what we're doing though.

After doing the xml sitemap test on the Knowledgebase [http://kb.madcapsoftware.com/], we then generated a sitemap starting from http://kb.madcapsoftware.com/mobile/Advanced/, which includes the same knowledgebase content as that above but presents it in the mobile skin.

On the mobile output, our sitemap generator crawled a total of 294 pages. That's more like it! The skin used in Flare's mobile output is straight html with no frames. It's easier for Google to find the content displayed this way, which means it's easier for end-users to find the content, too.

[NorthEast]

We'll probably be putting our help online in the near future.

One idea that I had was to insert a topic in the WebHelp that contains the TOC proxy; so it will include a link to every topic contained in the TOC. Provided Google (and other search engines) are pointed to this topic, it should mean that the vast majority of the help topics would be crawled.

[jbrock]

Inserting a topic with the TOC in it is not a bad idea. The challenge would be helping the search engines to find that page.

Probably the best thing you can do is create an xml sitemap that list all the pages in your Webhelp that you want Google et. al. to crawl. This is also the lowest hanging fruit, fortunately. You can use this xml sitemap generator http://goo.gl/5S3kKF. I purchased the self hosted option years ago and have used it often.

Simply set the sitemap's starting page as that topic that contains the whole TOC. All the links from that page will at least get crawled, and at the end you'll have a sitmaps.xml file to upload to your site.

The other low hanging fruit you need to include is a robots.txt file. This is a simple text file that tells search engines what can and can't be crawled. I advise making this file and uploading it to the root of your domain before creating the sitemap. Otherwise, there may be pages in your sitemap that you don't want crawled.

The robots.txt is important for a number of SEO reasons, but I can think of a particularly urgent one for Webhelp. When you build Webhelp, as opposed to when you build a pdf, ALL of the topics in your project get processed through the generator. Unless you have explicitly conditioned out specific topics or directories, they will be part of your Webhelp output even if they aren't part of the TOC.

That's not good. Madcap should seriously consider changing that behavior.

Until they do, however, you should certainly condition out any content you don't want on the Webhelp. And, since the directory/file structure of the Webhelp section of your site will mirror exactly the directory/file structure of your content explorer, add the paths of the directories and files you don't want crawled to the robots.txt file.

You shouldn't manage the sitemap or the robots.txt files in Flare, though. Flare may insert superfluous xml. Also, Flare will not put the either file in the root of your domain, which is where search engines will expect to find them. The final directory structure needs to look like this:

• yoursite.com/robots.txt
  yoursite.com/sitemap.xml
  yoursite.com/../content/topic.htm

I'll send you a link to ours once it's published later this week, just so you have an example.

[NorthEast]

Great, thanks for that information - it'll come in very useful.

Unless you have explicitly conditioned out specific topics or directories, they will be part of your Webhelp output even if they aren't part of the TOC.
That's not good. Madcap should seriously consider changing that behavior.

Yep, I'm aware of this; it isn't an issue for us as we use conditions.

There are cases why you might not want to include topics in your TOC though; e.g. in our help, we have a number of topics that are used only for context sensitive help, or only for pop-ups; and we don't want to include these in the TOC.

The other low hanging fruit you need to include is a robots.txt file. This is a simple text file that tells search engines what can and can't be crawled. I advise making this file and uploading it to the root of your domain before creating the sitemap. Otherwise, there may be pages in your sitemap that you don't want crawled.

In Flare you can choose to not include a topic in the search results, which obviously wouldn't apply to search engines. I'd just have to generate a list of these 'excluded' topics and add them to the robots file; thankfully I don't have very many.

[jbrock]

In Flare you can choose to not include a topic in the search results, which obviously wouldn't apply to search engines. I'd just have to generate a list of these 'excluded' topics and add them to the robots file; thankfully I don't have very many.

Yeah, that's true for Flare's search engine .

If content is properly conditioned out before generating Webhelp, which you already do, you should be safe. The robots.txt file is a precautionary measure. Plus, you can link to your sitemap files from it, which may help some.

The REAL PROBLEM is getting Webhelp crawled in the first place. Keeping certain files from getting crawled is easy.

Is your help live? Could you send me a link to it?

[NorthEast]

Is your help live? Could you send me a link to it?

Not yet, we have a bit of red-tape to get through first; I'm just researching and preparing for what needs to be done now.

[jbrock]

Here's a link to our new webhelp section. Look around and see if it gives you any ideas.

[trimore]

I am glad I finally found this topic.... We are considering moving from a online document management solution to Flare. One of our biggest concerns is the integration or work-ability with google search appliance. The framed output worries me. Flares answer is the work around of using the xml sitemap. I am also concerned thought that the GSA presents the link in a way that it shows the webhelp correctly.

Jbrock
It also looks like you solved our other issue which is output that does not look dated in the frame format. At least that is our opinion.... Your output is very different. I wonder if we should be looking at mobile help output as well???

Is your menu at the top you own javascript and managed outside of Flare or is it the flare javascript heavily customized?

[jbrock]

So sorry for the delayed answer.

The menu at the top is a simply a Flare snippet. I simply followed the tutorial here - http://net.tutsplus.com/tutorials/html- ... down-menu/ - to create the dropdown, then turned it into a snippet and embedded it on my master pages. There's JavaScript required for the menu, but you don't have to add it to Flare's js files. You can just link to it separately in the <head> of your masterpage.

The only JavaScript I customized for Bomgar was that used in the custom toggles. See here where we show the license types and versions: http://www.bomgar.com/docs/content/bomg ... ofiles.htm.

All of the main webhelp was made with the Mobile output using custom masterpages. We then used Flare's default mobile output for the iPad guide: http://www.bomgar.com/docs/content/bomg ... ofiles.htm.

[i-tietz]

Most customers are going to start looking for help on Google, not in the search panel in Flare. You can have the best help on the planet, but you’ll only frustrate customers if they can’t find it on Google.

Honestly? I don't think so. I don't know anybody who would go to google to find the "official" help for some specific software or machine ...
F1 does the job and so does a click on the ? symbol or the ? menu or the context menu or the embedded help ...

Go to Google to look for help on how to insert concept keywords in Flare? IF for some really obscure reason I would want to do that, I would go to the URL where I find the WebHelp and search there ...
You gotta stick to realistic use cases and not compare help content with website content. Flare is no website generating tool but a help generating one. For websites there are better tools with less restrictions in that respect.
Everything you say applies to websites (and for that part I really agree with you!), but not to helps ...

[NorthEast]

Most customers are going to start looking for help on Google, not in the search panel in Flare. You can have the best help on the planet, but you’ll only frustrate customers if they can’t find it on Google.

Honestly? I don't think so. I don't know anybody who would go to google to find the "official" help for some specific software or machine ...

Well, I frequently use Google to find help on Microsoft products, because I know I will get good results, including 'official' content by Microsoft.

I don't think it's right to say most people will use Google or that nobody will use it; but I think a lot of people will try a search in Google (as well as in WebHelp), and whether they use it again will depend on their results. What you can bet on is that a number of your users will try Google, and they will probably will expect to find something. Although you can argue how valuable that is if you have access to the WebHelp and its search, it is probably a good idea to have some sort of visible online content for people who don't own your product and are researching it.

Also, as the current WebHelp search isn't actually that good when compared to most search engines, some users may try Google. Take this forum for example, the search feature is quite poor, so many people use Google to find forum posts. At the end of the day, people will use what works best for them, not necessarily what you want or expect them to use.

[i-tietz]

Nothing of Justin wrote so far can be proven. He's just stating things. It's his Opinion. Mine's different.

And:
Give up the hope to generate software (or help) that fulfils each and every expectation. "Some" users who go searching for help topics on Google cannot determine what we produce. There will also be
- "some" users who don't like to use the mouse and want to see the index or the search to be called via keyboard shortcuts or
- "some" users who prefer dropdowns to popups or
- "some" users who don't like long topics and prefer to jump to the next one until they're lost in hypertext,
- "some" user prefer long topics, so they can print them and have everything relevant in one go.
- "some" users would prefer a software (and help) that works under DOS
- ...

Last:
If Flare would leave the frameset and use divs and css instead, "some" users would complain about it not working correctly with the brandnew version of IAmTheUpAndComingBrowser or with the version from year dot.
You have to make decisions and you will ALWAYS leave a few users behind. As long as noone proves that it has become a relevant part of users behaving in a certain way or expecting something specific you're well advised to stick to what you have.
But I can also understand that somebody who makes a living from SEO thinks it's necessary.

[RamonS]

Well, I frequently use Google to find help on Microsoft products, because I know I will get good results, including 'official' content by Microsoft.

I agree with that, but this is for two reasons:
1. the search on microsoft.com is horrible, so bad that it cannot even find its own KB articles
2. documentation by third parties is often better and much easier to understand than the docs from Microsoft

In that regards, finding help on Microsoft products is a special case. For anything else I typically try a few things, then consult help, then google for it. If that doesn't help I look for help in forums and if it is really important to me I contact the official support. So local help is a prime source. Since this is about WebHelp it should not matter much if one uses the local search or Google. I'd try the local search first to eliminate getting all the eHow garbage (the connection between Google and eHow.com is blatantly obvious). But that's just me.

[LTinker68]

It also depends on how your customers use your software or whatever product you're providing help for. If your program is something that would be run on a device not connected to the Internet, for example, then the local help you provide would be extremely helpful to the end user. They might still use a computer connected to the Internet to search away from the device, but if they're working on the device then they'd want access to help there.

The priority should be trying to give your users all the tools they need to use your software. I think help provided with the software should be the company's top priority. That doesn't mean the author will think of every scenario or piece of info that needs to be added, which is why users often go to Google or other sites -- they have a specific situation that's not covered in the help and are hoping someone else has run into the problem. Ideally, the author should periodically look at some of the forums or blogs out there related to their software to see if there's something that needs to be added to or clarified in the help, but given the amount of work we all do, that may not be practical. So I don't see users going to Google as necessarily a bad thing (especially if your help is also on the web to be found), unless the info they get back is that people think your help sucks and isn't useful.

[jbrock]

But I can also understand that somebody who makes a living from SEO thinks it's necessary.

I'm implemented the webhelp when I was an employee. I assume the suspicious tone is due to the fact that I changed my signature yesterday. Oh well.

When we wanted to make the webhelp findable, our concern was usability, not marketing. Our stats showed most people beginning help-related searches on places like Google, rather than our site. Our webhelp needed to accommodate them. I'm sure this is true for many companies who provide webhelp.

I still think Google.com is the F1 button for the majority of people now, but you're right - I had not done statistical research to verify that.

[LTinker68]

I still think Google.com is the F1 button for the majority of people now, but you're right - I had not done statistical research to verify that.

I think in part it's a generational thing, but nothing says you can't take advantage of it. Just host your help on a public website and your help will be returned in the Google results like any other sites, forums, or blogs. If someone keeps using Google and your site keeps coming up and they keep finding the answers there, then they'll bookmark your site and keep coming back.

Unfortunately, Microsoft and other large companies deliver lousy help with their apps, so a lot of people go to Google because they're not expecting the app to have a good built-in help. I keep giving Microsoft a chance, but a lot of times I'll resort to Google because Microsoft's help doesn't have what I need.

[i-tietz]

When we wanted to make the webhelp findable, our concern was usability, not marketing. Our stats showed most people beginning help-related searches on places like Google, rather than our site. Our webhelp needed to accommodate them. I'm sure this is true for many companies who provide webhelp.

I still think Google.com is the F1 button for the majority of people now,

What sort of software are you producing that users have to open Google to get help? Your GUIs don't have a "?" symbol? or link "Help" or menu "?" or "Help"? Or is it the new development style not to offer CSH anymore?
Or do you offer help on your GUI and in spite of that the users search for help at Google? Then maybe you should rethink your help concept, cos' it's just like Lisa said: Local help has top priority - else: What are users supposed to do who don't have access to the internet, because e.g. they have sensitive data on their computer?

[RamonS]

Maybe it is a web app. Pressing F1 in a browser gets you the browser help....explainable, but utterly stupid in the era of the cloud.

[sdcinvan]

Here's a link to our new webhelp section. Look around and see if it gives you any ideas.

Major kudos to you, Justin. Your documentation page is the best example of this done right! I love it and it inspires my planning for our documentation page.

I am curious how you inserted the email forms (and the ? pop-up help call-out) into your Flare output. This is not a native Flare feature (wish it was). Any tips?

Best regards,
Shawn

[jbrock]

Thank you, Shawn. We recently rebuilt our support docs site. This time around, I used Bootstrap to build our masterpages. The modal with the email thingy is using Bootstrap + Feedburner. Feedburner basically takes the RSS feed from our changelog and sends an email to folks whenever it's updated.

You can play around with a sample project using this method. Download it at flarestrap.com. I haven't had a lot of time to post guides and updates to the project, but I would certainly love any feedback.

Thanks again!