I too had this problem. Chrome and Opera browsers no longer support Java applets, so nothing you can do there I'm afraid. Firefox almost works but you cannot view the camera image and draw a motion detection area. The only way I did manage to get this working was to use Internet Explorer and add the website to the Compatibility View Settings. I tried this on Windows 10 and it worked just fine.
I'm not sure how long Microsoft will continue with this feature in Internet Explorer, so it might be worth looking into running IE7 or IE8 in a virtual machine by way of future-proofing (see https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ for free VMs from Microsoft with IE installed).