Website Watcher - track and analyze changes to webpages on auto-pilot


Watcher is an automated tool for tracking and monitoring changes to web pages. Watcher will monitor and detect changes to page titles (title tag), page description (description meta-tag), first-level heading in a document (h1 tag), page content and HTTP status codes of pages.

Getting started with Watcher


How do I tell Watcher to scan particular parts of the website?


The Watcher tracks changes to title tags, description meta-tags, h1 tags and webpage content on auto-pilot.

In order to scan a specific part of the content (only textual content, one or several text blocks or certain parts of HTML code), add the tags into the source code of a web page:

<!--start_content--> - in the beginning of the content fragment.
<!--end_content--> - in the end of the content fragment.

Custom tags
Regular expressions (not recommended)
You may use regular expressions to tell Watcher which fragments of the content to scan.

For example:

<div id="content">(.*?)<\/div>

Please note that to enable Watcher to scan your content, you should add "class", "id", and other attributes to the tag. This way Watcher will be able to pick and scan only requested parts.

Regular expressions
You may use regular expressions only to mark specific part of the content, to be scanned. You cannot use regular expressions to tell which parts of the content Watcher should skip.

Regular expression example
If you don't have enough expertise to use regular expressions to tell Watcher which fragments of the content to track, use Custom tags to mark the content.


How do I tell Watcher to skip particular parts of the website?


Today most CMS support dynamic content that can change in response to different conditions: page views, ratings, votes, search history, product order, filters, status (online/offline) etc. In most cases, there is no need to track these changes.

To prevent Watcher from tracking dynamic content or any other fragment of the content, add custom stop tags into the source code of the page.

<!--start_content_off--> - in the beginning of the content fragment.
<!--end_content_off--> - in the end of the content fragment.

Stop tags
You can place several custom tags on the same page :

<!--start_content-->
<!--end_content-->

and

<!--start_content_off-->
<!--end_content_off-->


We recommend applying to a webmaster or other technical specialist, to make sure you add tags in the correct way.


Pages for scanning


To get started with Watcher, follow the below instruction:

  1. Create a project or pick one from the list.
  2. Move to Watcher page and switch to the Settings.
  3. Add one or several pages to scan List of pages to scan or import pages from the sitemap. Import mask matches from sitemap Use * or / to import all pages from the sitemap. Import pages from the sitemap
  4. Press OK.


After you set up a list of pages, press Scan, to run the first check and record original data in the database.

Run Watcher

Wait until the tool scans your pages, and make sure the check has completed successfully. Please note, that during the first check, Watcher updates details for all pages you added. In the example below, Watcher updated all 96 pages.

First check report

The next time you run a check (on demand or on schedule), the tool will find all changes to the webpages and alert you of them.

Changes spotted
Please note, that the column Value contains current data, and the deleted strings in the column Changes are highlighted red. This way you'll alway be aware when someone deletes data occasionally or on-demand.

How do I schedule automated Watcher reports?


To schedule automated Watcher reports, follow the below instruction:

  1. Move to the Settings.
  2. Schedule reports. Watcher schedule

How do I manage Watcher alerts?


To manage Watcher alerts, follow the below instruction:

  1. Move to the Settings.
  2. Pick an option: send alerts to Notification center and/or to Email. Manage notifications

You account Email address will be pulled and added to the report automatically. If necessary, you may add Email addresses of other Topvisor users, to send a copy of a report to them.

Watcher in server logs


Each time Watcher visits a website, it sends a web server its details. This information is a part of an
HTTP query, that looks as a text string with User-agent: or User-Agent: in the beginning:.

Watcher will look as this User-Agent: Mozilla/5.0 (compatible; TopvisorBot/1.0)