http://billing.hostww.net/announcements.php Fri, 01 Aug 2008 13:03:20 EDT http://billing.hostww.net/announcements.php?id=11 Fri, 01 Aug 2008 00:00:00 EDT Tonight HostWW along with the majority of all Orlando located hosts and atacenters experienced a network wide outage due to a core router at the Level3 head facility failing.  Level3 technicians were instantly alerted and dispatched to replace the faulty hardware. At no time was our datacenter or servers actually shutdown or experiencing issues, rather the datacenter and citys primary backbone and infrastructure was inaccessible. We will post a full review once more information is received.  At this time the majority of all network connectivity should be restored.  Please let us know if you're still experiencing issues "Level3 is a transnational fiber backbone provider utilized by the top industries and enterprises in the world.  More information can be found @ http://www.level3.net/" http://billing.hostww.net/announcements.php?id=12 Fri, 01 Aug 2008 00:00:00 EDT As you may of noticed this morning starting around 3:30AM we had multiple BGP Sessions drop with one of our upstream providers (Level3). This caused some instability in the network while routes were re-routed. This issue was also compounded by a large DDoS attack targeted at our core networking system. As a result of the attack, troubleshooting of the initial route related issue was made much harder, thus extending time to get things sorted out. The network team was able to resolve the issue completely and everything should be running now. We will be looking into strengthening our internal policies to help alleviate issues such as these in the future. Thank you for your patience during this time. http://billing.hostww.net/announcements.php?id=10 Sun, 27 Jul 2008 00:00:00 EDT A few days ago on July 25th we experienced a massive DNS outtage from approx. 11:15am until 12:45am.  This only effected those clients who utilized our nameservers such as shared and reseller hosting account.  Generally all VPS and dedicated clients utilize their own servers for DNS.  I'll outline below exactly what happened and what was done during the outage to resolve the issue. Currently our servers are setup only to auto-update the latest security packages and updates as well as updating CPanel itself within the 'STABLE' build releases for CPanel.  No updates outside this stable build release are auto-applied, and we never upgrade or go beyond CPanel's 'RELEASE' build type. (There are two other builds CURRENT and EDGE which have the latest and quite unstable additions/upgrades.)  There are times when a bug or major issue is found with a recent STABLE build that we do upgrade to a RELEASE build for a temporary amount of time until the issuse are fixed within the STABLE build.  The night of the 24th and early AM on the 25th was one of these nights as there has been a major issue with 'dnsadmin' found to be taking 99% of cpu and slowing servers down during DNS zone syncs and changes made to clustered DNS servers. The upgrade for CPanel completed properly and we did not at that time see any issues. Also during each night all of the DNS servers within our cluster do a full sync, and before this issue occurred we had never experienced any issues relating to the syncing of all the DNS zones. First thing in the morning we began noticing a large amount of corrupted *.work.db files within our /var/named folder.  These work files are what the 'dnsadmin' process uses as it syncs the latest data between servers, and our nameservers had stopped resopnding to DNS requests and were showing up as 'lame' also known as non-authoriative. (This was the beginning of the downtime.) At this time we de-clustered every server to avoid any further propegation of the corrupted DNS zone files, and began to downgrade the servers back to the STABLE build as we were unable to directly get ahold of CPanel developers at this time to verify whether or not the upgrade has caused the corruption, though this was most likel the cause. We also contacted the NOC (network operations center) and began working with their tier three admins to resolve this issue and get our DNS zone files cleaned as quickly as possible.  There were 3 admins at the NOC as well as myself and one remote admin working on the issue to get it back up ASAP, though with 2000+ DNS zone files it did take some time. In addition to the corrupted zone files we also rebuild our named.conf (the configuration file for the named/bind service) to restore the nameserver's to their authoritive non-lame settings so they would answer as the primary response servers for each DNS zone. By 12:45pm EST this work had been completed and our DNS services were fully restored.  to avoid this in the future we have setup a seperate nameserver that does not run any other services and also keeps better records and backups of the DNS zones and named.conf as well as written some scripts to check and notify us as soon as even a single DNS zone is corrupted.  As this was a combination of issues that lead to the downtime we do not forsee any future problems witht he DNS cluster. Thank you for your patience during this time and continued business with HostWW. http://billing.hostww.net/announcements.php?id=9 Fri, 25 Jul 2008 00:00:00 EDT DNS outages began around 11:15 AM EST on 7/25/08.  The primary issue was resolved at 12:45 EST.  The outages were due to a problem with the CPanel update on the servers.  Our datacenter staff is working on the issue currently.  Our lead administrator will update this with a complete write up of what problem existed and what steps are being taken to make sure this does not happen in the future. http://billing.hostww.net/announcements.php?id=8 Fri, 18 Jul 2008 00:00:00 EDT We apologize for the delay in sending this out, per the power issues we ran into May 23 2008 where we mentioned that things were being done to be pro-active on ensuring no such outages occurred again, we had scheduled a planned UPS / Battery maintenance to replace / add some things to our systems. This was scheduled to be done Saturday July 12 2008 between 6AM EST and 10AM EST. There was no public announcement made about this because this maintenance was not to effect any clients or internal systems. However just in case, we planned it on a weekend during early morning hours in case something did go wrong we would be on standby and being the weekend the least possible impact it would have on clients. Shortly before 10AM est when the maintenance was about to be completed there was what we thought at the time a DNS network wide issue which we thought effected some servers in the data center. We noticed the call volume increase and immediately began to investigate the issue. We advised staff taking calls to let everyone know it was an intermittent connection issue as we didn't know what the real cause was just yet. Shortly after that we noticed that the EATON UPS maintenance staff had completed the maintenance and attempted to move our entire DC Floor load back to the UPS. The whole data center runs off 3 phases, when the Eaton UPS staff switched the load to the UPS one of the 3 phases failed to fully switch and caused over 33% of the servers in the data center power cycle. It was so sudden and so fast we initially thought it was a DNS issue and thus why it was said initially as a possible network issue when in reality it was one of the 3 phases failing to fully switch over to the UPS. During this time there was about 2-3 intermittent loss of power to the 3rd phase of the load thus why some of you saw your servers go on and off more then once. We immediately went to the Eaton Power UPS staff and advised what happened as this maintenance was not to suppose cause any service interruption to any clients. One of the tasks of the maintenance was to replace the capacitors in the system. The UPS system has over 50 of these capacitors which provide power via 3 electrical phases to the entire NOC, he immediately admitted that one of those capacitors was not wired correctly from the factory thus why 33% of the DC load power cycled on and off during attempts to move the load to the UPS as it shorted it out the phase. They immediately corrected it and shortly after that the entire NOC power load was fully running on the UPS. Eaton is a public company http://www.eaton.com/ and we are shocked how this happened and are determined to get to the bottom of it. The situation is still unacceptable under no circumstances but out of our control. Our culture is built upon integrity and everything stated is 100% true. The only good news that can come out of this is that all the needed maintenance/changes/upgrades to our power systems is completed 100% and we are looking forward to the next 5 years ++ of uptime We take downtime seriously and have acted pro-actively to ensure you get 100% uptime each and every single month. Services have remained 100% since then and are expected to remain 100% from here on out. If there is anything we can do for you please just reach out to us and we will make it possible. We are determined to make your business relationship with us something you can count on with peace of mind. We put our entire soul and mind into this company with a desire to give you the best level of service and support. We are at your disposal at anytime.During the recent downtime experienced this past weekend our HostWW.net primary website as well as our client support/billing site was offline.  The site was being hosted on a private server that was setup with redundant hardware and network connections, though our main site was unfortunately not clustered with an off-site provider as it should have been.  As this off-site clustering was overlooked with our previous upgrade we highly apologize for the in-ability to reach our support staff and primary site during this past weekend’s downtime. In reference to this we are going to be setting up our main site as well as our client billing and support sites in multiple datacenters in a fully clustered and redundant configuration to ensure that this will never ever happen again.  This will be done transparently  and there will be no downtime during this internal upgrade.  Please let us know if you have any questions and concerns or recommendations.  Thank you for your continued support and business with HostWW, and we look forward to providing you with an even greater level of service excellence in the future years to come! http://billing.hostww.net/announcements.php?id=7 Mon, 26 May 2008 00:00:00 EDT Recently HostWW and some of our most dedicated clients both resellers and shared hosting accounts alike have been attacked time and time again by spammers. These attacks have been contained by both the use of our automated security as well as our personal intervention and manual blocking. In order to provide the most secure and stable hosting enviroment we have recently made a couple of emergency changes to our apache/php configuration. These were to be planned upgrades within the upcoming weeks and have been going through testing stages, though with recent attacks we decided to initiate them now. Our apache daemon has been upgraded from v1.3 to v2.2 in order to provide better memory and thread handling as well as the latest security patches. Along with this apache upgrade we have also added the security of suPHP. More documentation on suPHP can be found below, but in short this stops php scripts from running on the server as "nobody" and instead runs all php scripts as the owner's username. This makes tracking and blocking spammers as well as rogue php scripts 1000x's faster and easier. Since suPHP is now enabled we have also enabled the CPanel security tweak to dis-allow any email sent from php scripts using the system user "nobody." More information on this can be found below. 98% of our customers sites and scripts will be totally uneffected by the backend changes, and we have done our best to scan and test customer sites for issues, though if you experience any problems with your site or scripts sending email please contact is and we will resolve any issues with top priority. Thank you for your time and continued business with HostWW. If you have any questions/comments or concerns please feel free to contact us. Have a spectacular day! (suPHP Info) http://www.suphp.org/Home.html (Under the Mail Settings - disable nobody sender) http://www.cpanel.net/support/docs/11/whm/server_config_tweaksettings.html Here to Serve, The Staff @ Host World Wide http://billing.hostww.net/announcements.php?id=6 Fri, 23 May 2008 00:00:00 EDT Effected clients, co-workers, partners: At approximately 8 A.M. EST a segment of our data center experienced a power outage. Shortly after that we realized the city had experienced a power outage as well which attributed to the loss we saw in the NOC. We immediately went into generator power and we are still currently running on generator. What happens when there is power loss is that we immediately go into UPS for a few minutes meanwhile the generator powers up. The UPS ensures no service disruption meanwhile the generator turns on. This happened normally but approximately 40% of the servers tripped during this process. Power loss and going into UPS followed by generator power has happened a few times in the past. But we have never experienced what has happened today - the power surge during this outage was extremely high. We are meeting with our electrical engineers as we speak to analyze the situation to learn exactly what happened. We still have the data center running on generator as the power in the city could still be unstable and we do not want to risk any further outages. As many of you know these kind of issues do not happen often and we are taking this incident very seriously. The data center is equipped with a high level of UPS and generators are in place to ensure maximum uptime. We apologize for the inconveniences as we know this has effected a lot of you and your clients. We will ensure we find out in detail what happened, resolve it completely and put measures in place so it doesn't repeat again. All effected servers should be operating normally now and we sincerely thank you for your patience.HostWW.Net Support Department http://billing.hostww.net/announcements.php?id=5 Tue, 25 Mar 2008 00:00:00 EDT You may now transfer or register a name using HostWW.net and it will cost only $8.49/yr.  Once your purchase this domain is is yours for as long as you keep renewing its registration each year.  Buy a domain today and get a hosting account free for a month. http://billing.hostww.net/announcements.php?id=4 Wed, 12 Mar 2008 00:00:00 EDT VPS is now ready for ordering and instant activation!  Visit our VPS page and see what we have to offer for you at http://www.hostww.net/dedicated.php http://billing.hostww.net/announcements.php?id=3 Thu, 14 Feb 2008 00:00:00 EST We have added three new servers online this week as well as upgraded memory in  some servers that are already online. This would be the cause of some server restarts. http://billing.hostww.net/announcements.php?id=2 Sun, 03 Feb 2008 00:00:00 EST HostWW is always making changes to better serve our customers!  We have 24/7 Technical Support available by phone and e-mail!  You can reach our 24/7 Customer Service Center at 1-866-355-7723.  You can always e-mail us! Creating Support Tickets has never been easier! Now all you have to do is send a e-mail to support@hostww.net or billing@hostww.net and your support or billing ticket is automatically created, and assigned to your account as long as you e-mail it from the address you have registered with us! Refer Clients & Make Money! You can refer clients to HostWW easily and make money!  When you login to your account at [ClientAreaLink] and click Affiliates -> Activate Affiliate Account, you will be signed up to be an affiliate and make $5 for each customer you refer to us.  You will have a special ordering URL to give people and it saves a cookie in their browser so if they return to the site later and order we will know to give you credit. Use HostWW Automatic Bill Payment! Login at [ClientAreaLink] and click My Details -> Change Credit Card Details.  Once you update it with your credit card information your services will be billed to that card each month. Manage your Account Contacts You may now login and manage contacts for your account!  You can add additional contacts to the account and customize them to receive general , product, domain, invoice, and support emails.  Of course adding these preferences also sets what details they will be given when they contact HostWW support!  You can do this by going to [ClientAreaLink] and logging in clicking My Details -> Add New Contacts. Check the Status of Our Servers Anytime! Login to your account at [ClientAreaLink] and click the Support Tab -> Then Click Server Status!