UPDATE: 3:19 AM GMT

From the datacenter:

Service Restoration Announcement

We are thrilled to announce that our services have now been fully restored! After dedicated efforts and hard work from our team, everything is back up and running smoothly.

We appreciate your patience and understanding during the restoration process. Your support has been invaluable to us.

If you encounter any issues or have any questions, feel free to reach out to our support team.

Thank you for being part of our community!

Time posted

Jan 17, 20:19 MST

UPDATE: 2:14 AM GMT

We are seeing intermitting access but most systems are recovered.  Sites may be slow or connections drop.

PNAP Update:

Update - We're making significant progress in restoring services. As things return to normal, we appreciate your ongoing patience. Our dedicated team is working hard to ensure a seamless experience, and we'll keep you posted with further updates. Thank you for your understanding.
Jan 17, 2024 - 19:04 MST

UPDATE:  12:58 AM GMT

Most servers are online but we are still seeing packet loss and some services not responding.  Web services appear to be mostly restored. 

PNAP Update:

Update - While services are still undergoing restoration, your patience is invaluable as we navigate through the final steps of the process. Please note that intermittent issues may still persist, but rest assured, we're actively addressing them. We sincerely appreciate your understanding and assure you that we're working diligently to bring everything back to full functionality. Thank you for your continued patience.
Jan 17, 2024 - 17:34 MST

UPDATE: 12:09 AM GMT

We are seeing servers come back online but do not have an update from the datacenter at this time.

NOTICE: 11:15 PM GMT

Server located in the PhoenixNAP facility in Phoenix, AZ are subject to an outage.   We will update customers as we have more information.

From the data center:

We are currently experiencing a site wide outage of all services in our Phoenix Facility, we will provide further updates as we receive them from our investigating teams.

https://status.phoenixnap.com/

cPanel/WHM License Price Changes

cPanel has announced a cost increase in their licensing.   rackAID includes licensing prices as part of your server pricing.   Where possible we will absorb the cost of the license cost increase.   However, for customers with discounted or entry level servers, we will have to adjust your pricing to offset the price increases.   This will range from $5-10/mo depending on the number of accounts on your server.   These price increase is effective starting Feb, 2023.

Cloudflare

Cloudflare has announced a $5/mo price increase on all Pro plans.   This price increase is effective starting April, 2023.  All customers with Cloudflare Pro, will see their price increase to $35/mo for the service.   This includes the $25/mo Cloudflare fee and a $10 management fee

Plesk

Plesk is also increasing licensing costs.   We have been able to absorb most of the cost increases but for some customers this may translate to a $5-10/mo price increase.    The price increase is effective starting Feb, 2023.

If you have any questions about these changes, please let us know.   We will try to contact each customer directly about upcoming cost increases.

A vulnerability has been discovered in Apache Log4j, a very ubiquitous logging package for Java. Successful exploitation of this vulnerability could allow for arbitrary code execution within the context of the systems and services that use the Java logging library, including many services and applications written in Java. Depending on the privileges associated with these systems and services, an attacker could then install programs; view, change, or delete data; or create new accounts with full user rights. If these systems and services have been configured to have fewer user rights, exploitation of this vulnerability could have less impact than if they were configured with administrative rights.

Ref: https://www.cisecurity.org/advisory/a-vulnerability-in-apache-log4j-could-allow-for-arbitrary-code-execution_2021-158/

rackAID does not use any Log4j containing services as part of out management package installed on your server.

How does this impact me?

If you are not using Java-based applications, then you are not impacted.    No further action is required on you part.

If you are using Java-based applications, check your application to see if it includes the Log4j library.  If yes, follow your software vendors guidance on remediation.

cPanel Solr Plugin

The WHM/Cpanel includes an optional Solr search plugin for IMAP email.    This plugin contains the impacted Log4j library.   cPanel has already released a patch for this plugin, but to be safe rackAID has deleted this plugin from all of our cPanel servers.  If you were using this plugin, please let us know and we will re-install it.

ElasticSearch/Solr

These two search tools may be used with some discussion forums.   We have contacted server owners where he have found this software installed.   Sphinx search does not use the Log4j library, so it is not impacted.

Java/JVM

rackAID has scanned servers for any java/jvm processes.   We have contacted server operators if we suspect their applications may be vulnerable. 

rackAID continues to monitor the COVID-19 pandemic. We will act according to national and local guidelines regarding quarantines and best practices.   

As with all IT businesses, we rely on many vendors, service providers, and support teams. We have received business continuity statements from our major infrastructure providers (IBM, AWS, and PhoenixNAP). They have implemented safeguards for their employees as well as assured that services should continue as normal.

rackAID support staff will be working remotely and from home. We have assured that all staff also have redundant internet connectivity from home. They can use our VPN or secured SSH gateways to reach impacted servers securely. At this time, we do not anticipate any service interruptions due to COVID-19.

Firewalls, IDS/IPS

If you are now working from home and cannot reach your services, we may need to make a firewall adjustment. Please open a ticket with the subject of "Firewall Changes." Our team will work with you to add your IP to the firewall. 

We protect our networks with various Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS). Thes systems may automatically block your IP address if you trigger detection rules. If you can no longer access your services, please open a support ticket and include:

• Your IP address
• Your user account
• The service you were trying to access (SSH, FTP, Email, HTTP).

Our team can then remove the block and whitelist your IP. 

We will also be requiring additional identity verification for any service requests that impact security, user credentials, or substantially alter operations. 

All of at rackAID, wish you and your families a safe and healthy journey through this challenging period. 

Jeff Huckaby

CEO, rackAID LLC

We protect our operations with Cloudflare's Web Application Firewall service.   Recently, we have added new rules that block some code, SQL and error messages.   

If you try to post a ticket and Cloudflare blocks your submission, please move any code or other data into a text file and attach it.  

Alternatively, you can use a service like https://www.pastebin.com/.    We use this service to share logs and error messages.  Set it to be private and expire in a reasonable period.  I use 1-7 days for non-sensitive information and hours for sensitive details.