Troubleshooting suspected hacked sites

One of the most popular features of Watchful is the Early Warning Audit that monitors changes in critical system files. Unexpected modifications of these and other sensitive files may be an indication of unauthorized access. 

If your web site has indeed been hacked, modifications to these files are not likely to be the source of the hack. Rather, your site is likely to have been hacked elsewhere, and the hack was then used to modify some of your sensitive files.

Thus, we recommend that any unexpected modification detected by Watchful be followed up by a full investigation.

As a general outline, we recommend the following workflow when investigating unexpected file modification notices that you receive from Watchful.

Please note that if you are not confident in your ability to compare large file sets (as described below) or if this article sounds confusing or is hard to understand, we recommend that you hire a professional to investigate changes in your site.

1. Enquire with your coworkers, colleagues, contractors, employer or anyone with access to your web site and/or your web server to determine if someone has made the change without your knowledge. 
   
   
2. If you still suspect unauthorized access following Step 1, disable public access to your web site by enabling HTPASSWD protection — sometimes called directory protection — on your entire site during the troubleshooting steps below. You may need to contact your hosting provider for help setting up an HTPASSWD, though it is a common feature in most hosting control panels.
   
   
3. Create a backup of your site using your backup solution of your choosing.
   
   
4. Compare or DIFF the backup created in Step 3 to your most recent backup (ideally, these are stored at an offsite location). Many text and HTML editors will include a DIFF feature, but as of this writing we recommend DiffMerge since it is free and runs on Mac OS, Windows and Linux. 
   
   
5. Repeat Step 4 with progressively older backups until the file noted in the Watchful notification email is identified as different between the two backups. By noting the date of the backup that first locates the file change identifed in your Watchful notification, you can determine the earliest possible date on which you site was hacked.
   
   
6. Once you have determined the earliest date of the hack (Steps 4 and 5), examine all the files added, removed and modified between that date and the backup you created in Step 3. For each of these files, ask yourself if the changes are expected or not, and also DIFF each individual file. For PHP, javascript, CSS and HTML files, this will allow you to examine all the changes that have been performed. 
   
   Note that on active sites you might find a lot of new images between the old backup and the most recent one. These may be completely normal and expected, but also note that corrupt images can also be used to exploit some vulnerabilities. You may also find scripts uploaded to your site.
   
   
7. With some luck, after all this you will have found a source of the attack, perhaps a file or script that provides a backdoor for hackers. However, it may not be clear how this backdoor was added to your site in the first place.
   
   If the hacker has not carefully hidden their tracks, you may be able to use the filename of the backdoor to search the server logs (obtained from your web host or perhaps your web hosting control panel) and further determine what requests were sent to the server that allowed the backdoor to be uploaded. This may then reveal the true source of the hack, a vulnerability in plugin or extension for example.
   
   
8. Based on the results of your investigation, take any and all appropriate action to (i) remove any dangerous files added to your site, (ii) restore hacked or deleted files to their original state and (iii) patch the true source of the problem.
   
   You may also want to repeat the entire process using the very oldest backup available (preferably from the day your site was launched), but for active sites this task quickly becomes time and cost prohibitive.
   
   
9. Once you are satisfied that you have found and removed all the hacked files and backdoors, it’s time to re-publish your site by disabling HTPASSWD protection that you enabled in Step 2.