Sentiment Radar is fully online and working. The service has been fully restored. As a user you can again analyze new products and add custom features to existing ones.

The problem with the link between the processing service and the front end service resulted from a combination of things. One which we control and one that we sort of don't.

First, what we don't control. Our deployment system relies on a continuous integratin tool provided by a 3rd party. This tool is currently having trouble correctly downloading code to execute. This seems to be the result of a recent change by the 3rd party, we have been in contact with them and are tracking their investigation.

By sort of don't control I mean to say that we don't control that software. However, we have choosen to use it. So in a way we most certainly do control this aspect of the problem.

Next, what we do control. It was late a night when I went to deploy the new changes and I had to manually since the automated system isn't working, as above. In doing so I messed up a few settings during the deployment and didn't set the link between the processing engine and the front end up correctly. In a bit of a panic to get things back to live that night I made changes that I shouldn't have to the deployment system which only made matters worse.

Lesson 1: don't run deploys late at night. Be fresh and clear headed.

Lessing 2: rapid changes to the deploy system late at night are never the right answer.