Hi, this is Low power writing. There was a service outage started at 2024-09-15 12:59 UTC, for both Rivoreo main website and my personal website. I'm going tell the details about this incident.
Here's the story.
Some of our websites are hosted on the Rivoreo Tokyo server (104.238.151.234), a cloud virtual instance provided by Vultr.
In 2024-09-16 (+08:00) I received an error report from Google that indicating something has gone wrong with https://rivoreo.one/ site; I then realized the Rivoreo Tokyo server no longer reachable. Immediately opening the Vultr account web, I found the cloud instance was stopped, and this suspicious-looking ticket in the Support tickets list:
http://johnny.moe/? That's the personal website of my friend Johnny Sun; which is of course, hosted on this server too. The truly surprising part however, is the file in this report, nc.exe.
Back in days of years 2016, when I still routinely work with other people who may have issues on their computer that runs Windows. Dealing with this operating system is quiet challenging because it lacks many diagnostic tools that I'm familiar with in Unix-like systems. To make this easier, I put some Windows builds of these basic tools into the only public web server I had at the time. They are being put into the default website, so I can download them by using the numeric IP address, 104.238.151.234 (Yes, I remembers the numeric IP address perfectly well; I can type it faster than most domain names) directly. When Johnny Sun's personal website being set up, it simply used the default website configuration, thus inherited these old files.
The error in this report is that the file in question is a Windows build of the traditional Netcat, a popular networking tool; often used for network diagnostic purpose, and is of course, a legitimate program.
Granted it is possible for some bad actors to abuse Netcat for malicious activities, but this didn't make the program itself malware. In fact almost every piece of legitimate software can be abused in one way or other; flagging them as malware based on this is obviously absurd.
I tried to start the instance from the web panel, but apparently there's issue with that action. I'm not sure whether this is due to a bug in the panel, or intended restriction.
I leaves a message in the ticket right away, stating that the program isn't a malware, as well as the issue I encountered for starting the instance.
After performing some necessary configurations for other servers, in response to the unavailability of Rivoreo Tokyo server, I turned my attention onto the URLhaus page, which looks like this for the time being.
I filed a false positive report here.
By this time, I began to take proactive and aggressive countermeasures to defend my other public-facing websites, including the newly opened source browsing site for Rivoreo, from this particular probe, by setting up a newly written CGI program to serve the /nc.exe path; more information at end.
With the Vultr support ticket still gets no reply for hours, I tried their API to start the instance; to my surprise, that actually worked; the operating system started quickly and I entered the system. I immediately began to install the same CGI program into the web server software, as well as to review the access logs of every website hosted there.
This takes some time. In reviewing of the access log of http://johnny.moe/, I got these suspicions accesses:37.19.221.171 - - [15/Sep/2024:12:19:48 +0000] "GET /nc.exe HTTP/1.1" 200 106682 "-" "Wget/1.24.5" 37.19.221.171 - - [15/Sep/2024:12:33:09 +0000] "GET / HTTP/1.1" 200 1487 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" 37.19.221.171 - - [15/Sep/2024:12:33:10 +0000] "GET /assets/css/main.css HTTP/1.1" 200 3233 "http://johnny.moe/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" 37.19.221.171 - - [15/Sep/2024:12:33:11 +0000] "GET /assets/css/images/overlay-pattern.png HTTP/1.1" 200 14186 "http://johnny.moe/assets/css/main.css" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" 37.19.221.171 - - [15/Sep/2024:12:33:11 +0000] "GET /assets/css/images/overlay.svg HTTP/1.1" 200 1105 "http://johnny.moe/assets/css/main.css" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" 37.19.221.171 - - [15/Sep/2024:12:33:12 +0000] "GET /favicon.ico HTTP/1.1" 404 499 "http://johnny.moe/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" 37.19.221.171 - - [15/Sep/2024:12:33:11 +0000] "GET /assets/css/images/bg.jpg HTTP/1.1" 200 238323 "http://johnny.moe/assets/css/main.css" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
The first entry is the first request made from 37.19.221.171 in the entire server, suggesting this origin tested and hit the path on their first try on the website. Based on my speculation, the scenario is very likely the following:
A possibly inexperience 'penetration tester' runs a script to find suspicious file on random websites.
The script hits Johnny Sun's website, and downloaded a file named nc.exe.
A while later the tester finds the downloaded file, and opens the main page of the website in a web browser. The tester then concludes that the file is probably not the intended content of the website. But instead of contacting the site owner for this potential issue, the tester made a really bad move, by reporting the site to URLhaus, with an absurd 'malware' tag for this URL.
Because this is Johnny Sun's personal website, the contact information of him is readily available on the very web page that the tester has just opened.
Upon receiving the report, URLhaus did some automated analysis, then generates an automatic report to the IP address holder, Vultr. Vultr in turn, didn't conduct much investigation on their own, but proceeds stopping the corresponding instance almost immediately, without giving its customer any chance to avoid service interruption.
As the basic services recovery done, I updated the support ticket on Vultr, by leaving a new message:
Keep in mind when I finished those, and got the server secured from this particular 'attack' (I calls it an 'attack', shouldn't I?), it is already midnight in my time zone. It may not look like such, but the whole process is very time-consuming.
By reviewing the access log of this website, the very first access of /nc.exe in the entire past year is indeed from the tester, IP address 37.19.221.171. It is no way this legitimate, innocent program caused any harm to anyone, in anyway.
This irresponsible action took by that 'penetration tester' did not created any benefit for the internet, only let legitimate services being disrupted, and valuable time of server operators being wasted.
By the afternoon of next day (2024-09-17), the original support ticket filed by URLhaus on Vultr still gets no updates, I therefore opened another ticket for a compensate of any sort:
While waiting for replies from Vultr for both the tickets, I got an update from abuse.ch, which handled the false positive report I filed early from URLhaus. They have conducted a manual review of the original report, and concluded that the report is indeed false positive. The original URLhaus page for this report is now gone.
A screenshot of their response is available here.
I proceeds updating the original ticket on Vultr (which by the way still has no reply from Vultr), to include the link indicating that the original abuse report on URLhaus has been removed:
Shortly after than I got an unexpected reply. The Vultr support representative seems completely ignored my previous messages, and asking me to remove the 'malware':
Did they actually read my messages before posting this?
I posted my message to against each of theirs statement:
After a few messages in this ticket, the Vultr security team is no longer interested in resolving the 'malware issue'. You may read the full conversation in this ticket here.
In the other ticket however, things start to turn irreversibly wrong. After I asked the 'demand of compensate' a second time, the Vultr customer support team seems to have a discussion for a little while, then rejected it:
So the argue started. They even insist that there was really a 'malware' on my instance:
And they still reference the false malware report, despite a clear false positive conclusion has already been made for it:
What actually makes this post funny is, VirusTotal shows a small percentage of vendors flagged for this URL as malicious, is simply because URLhaus did so first. As misdetections and false positives are common in anti-virus products, VirusTotal uses as many as possible of similar products from different vendors to provide a combined view; this avoided relying on a single product or too few products, that may provide inaccurate result. For this reason, it is fairly easy to submit a clean program or URL to VirusTotal and trigger at least one of the vendors to raise malicious flag.
A screenshot of this VirusTotal page as of 2024-09-18 is available here.
I replied my point of view again, and pointed out that one of their previous statements was wrong: that file is still stored in original place of the instance, I haven't unlink(2)ed it, and the URL is still accessible, it will give HTTP status 200 rather than 404:
The final message from Vultr:
At this point I no longer hold any reason to stay on this platform, so I posted my final message on the Vultr Support site, as a Vultr customer:
Like I said there, this instance has been active for 9 years, along with its IP address. This was a pretty stable hosting service indeed. I has been so used to type this IP address over these years; it makes me sad to see it go. On the bright side, I feels good knowing my servers are now automatically blocking those irresponsible 'penetration testers', including the newly consolidated Rivoreo Falkenstein server, which now hosts all services that previously hosted on Rivoreo Tokyo server.
Vultr just closed that support ticket without leaving a reply. The full conversation of the ticket is available here.
So that's the end of the story. I'm going back to work on some software project, which is at this moment, RNCN node services.
To help identify and block those bad 'penetration testers' in the websites I operates. I have installed a CGI program for all of these websites that could be reached publicly on internet. It currently serves the /nc.exe endpoint; but is subject to be extended to cover other similar endpoints in future. This CGI program will do one of, or a combination of the followings:
To avoid triggering any of the above action, I strongly advise anyone to not attempt accessing this path on any website that operated by Rivoreo.
For anyone want to protect their website from similar threats, I also suggest putting the forbidden paths into /robots.txt; so any bot accessed these paths can be flagged as not respecting the Robots Exclusion Protocol. For example:
user-agent:* disallow:/nc.exe$
