[Dec 28, 2001, 1:03 PM]
Network Problems
Around 1pm Eastern time today, we have experienced an internal network
problem in which our Foundry switches fought each other for control of
network traffic. We believe this is due to a bug in the Foundry software,
but no fix has yet been identified. Network traffic was disrupted for
approximately three minutes.
Beginning at 3pm today, we will be reconfiguring our network to eliminate
our reliance on Foundry-specific features which we believe are related to
this problem. No further downtime is expected.
[Dec 27, 2001, 6:05 PM] tama Maintenance Completed
We have completed the hard drive swap on tama (www70). Total downtime was
under 10 minutes.
[Dec 27, 2001, 1:59 PM] tama Emergency Maintenance
We have detected errors in the hard drive for tama.pair.com (www70).
We will be taking down this server on an emergency basis for a hard
drive swap. There will be two periods of downtime, each of which
should be no longer than 10 minutes. We will post additional information
when it becomes available.
[Dec 27, 2001, 9:32 AM] foma Downtime
foma (www176) rebooted under a high load. The downtime was under 10
minutes.
[Dec 22, 2001, 6:36 PM] chi Downtime
Ran out of swap space and was rebooted. Downtime was about 10
minutes.
[Dec 22, 2001, 3:30 PM] dieci Downtime
Rebooted due to high load. Downtime was less than 10 minutes.
[Dec 21, 2001, 12:24 PM] vert Maintenance Completed
The chassis swap on vert (www102) has been completed. The server
is now an Athlon XP 1600+, with 512MB of RAM. Downtime was 10 minutes.
[Dec 21, 2001, 11:58 AM] vert Emergency Maintenance
Due to several crashes within a short period of time, we will be taking
down vert (www102) in order to upgrade its chassis and RAM. Downtime
should be ten minutes or less. We will post additional information as it
becomes available.
[Dec 21, 2001, 11:38 AM] ynilo Downtime
ynilo (www166) was rebooted to clear up a problematic high load condition.
Downtime was approximately one minute.
[Dec 21, 2001, 8:53 AM] elhaz Downtime
elhaz (www144) crashed under load and was rebooted. Downtime was
approximately 15 minutes.
[Dec 20, 2001, 2:39 PM] agho Maintenance Completed
The RAM upgrade on agho has been completed. Downtime was less than 5
minutes.
[Dec 20, 2001, 2:30 PM] agho Planned Maintenance
In order to better handle increased load on this server, agho (www187) will
be taken down breifly for a RAM upgrade. Downtime should be minimal, and
the RAM boost expected to substantially reduce the load problems on this
server.
[Dec 18, 2001, 6:55 AM] delsi Downtime
delsi, www206.pair.com crashed under heavy load and had to be rebooted.
Total downtime was about 15 minutes.
[Dec 17, 2001, 4:23 PM] vert Downtime
vert (www102) crashed under a high load and was rebooted. The downtime was
under 10 minutes.
[Dec 17, 2001, 4:12 PM] flure Downtime
flure (www75) crashed under heavy load and required a manual reboot.
Downtime was 10 minutes.
[Dec 16, 2001, 5:37 PM] sowilu Downtime
sowilu (www145) crashed and was brought back online after a filesystem
check. Downtime was approximately 15 minutes.
[Dec 14, 2001, 7:05 AM] khin Downtime
The server khin.pair.com (www184.pair.com)
had to be manually rebooted. Downtime was
under 1o minutes.
[Dec 13, 2001, 5:36 PM] db11 Maintenance Completed
The hard drive swap on db11.pair.com has been completed.
Total downtime was approximately ten minutes.
This is the last hard drive swap for MySQL servers. None of
the servers are now running with the faulty IBM 75GXP model.
[Dec 13, 2001, 4:04 PM] db13 Maintenance Completed
The hard drive swap on db13.pair.com has been completed.
Total downtime was approximately ten minutes.
[Dec 13, 2001, 1:05 PM] db10 Maintenance Completed
The hard drive swap on db10.pair.com has been completed.
Total downtime was approximately ten minutes.
[Dec 13, 2001, 11:17 AM] db16 Maintenance Completed
The hard drive swap on db16.pair.com has been completed.
Total downtime was approximately ten minutes.
[Dec 12, 2001, 10:30 PM] ungwe Downtime
ungwe(www48) was manually rebooted this evening. Downtime was less than
15 minutes.
[Dec 12, 2001, 3:51 PM] db8 Maintenance Completed
The hard drive swap for db8.pair.com has been completed. Total
downtime was approximately ten minutes.
[Dec 12, 2001, 3:29 PM] db6 Maintenance Completed
The hard drive swap for db6.pair.com has been completed. Total
downtime was approximately ten minutes.
[Dec 12, 2001, 1:19 PM] db5 Maintenance Completed
The hard drive swap for db5.pair.com has been completed. Total
downtime was approximately ten minutes.
[Dec 12, 2001, 11:57 AM] db2 Maintenance Completed
The hard drive swap for db2.pair.com has been completed. Total
downtime was approximately ten minutes.
[Dec 12, 2001, 11:43 AM] foma Downtime
foma (www176) crashed under a heavy load and was rebooted. The downtime
was under 15 minutes.
[Dec 11, 2001, 3:41 PM] Planned Maintenance: db2, db5, db6, db8, db10, db11, db13, db16
During December, we are planning to proactively swap out eight MySQL
server hard drives, due to the high failure rate of the drive in use in
these servers. The above servers are using the IBM Deskstar 75GXP hard
drive model. We recently swapped out hard drives in shared servers for
the same reason.
Each swap will require two periods of downtime, each of which should be no
longer than ten minutes. The planned schedule is:
Wednesday, December 12
db2
db5
db6
db8
Thursday, December 13
db10
db11
db13
db16
[Dec 11, 2001, 2:30 PM] ynilo Downtime
ynilo(www166) had to be manually rebooted. Downtime was under 15 minutes.
[Dec 11, 2001, 8:39 AM] gao Downtime
gao(www111) had to be rebooted this morning. Downtime was under 10
minutes.
[Dec 10, 2001, 3:56 PM] csor Downtime
csor (www190) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Dec 10, 2001, 12:19 PM] db20 Maintenance Completed
The hard drive swap for db20.pair.com has been completed. Total
downtime was approximately 10 minutes.
[Dec 10, 2001, 10:27 AM] db20 Emergency Maintenance
We have detected hard drive errors on db20.pair.com. We will
be taking this server down for emergency maintenance. There will
be two periods of downtime, each of which should be no longer than
10 minutes. We will post additional information as it becomes
available.
[Dec 8, 2001, 9:49 PM] sether Downtime
sether (www95) crashed due to hardware problems. The hard drive was swapped
into a new server and is currently coming back up. Downtime was around 30
minutes.
[Dec 8, 2001, 5:20 PM] beith Downtime
beith (www17) was rebooted due to high load. Downtime was 10 minutes.
[Dec 7, 2001, 6:41 AM] merull Downtime
merull(www180) crashed under heavy load and had to be rebooted. Total
downtime was under 10 minutes.
[Dec 6, 2001, 3:58 PM] flure Downtime
flure (www75) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[Dec 6, 2001, 2:54 PM] vepar Downtime
vepar (www201) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Dec 6, 2001, 11:29 AM] zelaza Downtime
zelaza (www247) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[Dec 5, 2001, 10:27 AM] ceirt Downtime
ceirt (www39) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[Dec 4, 2001, 4:24 PM] anca Downtime
Severe load from a user program caused anca to be unreachable for nearly 20
minutes. The script in question has been identified and removed, and anca
has since returned to normal service.
[Dec 4, 2001, 3:45 PM] thurisaz Maintenance Completed
The upgrade of thurisaz (www136) has been completed.
Downtime was approximately 10 minutes. The server is now
an Athlon 1000, with 512MB of RAM.
[Dec 4, 2001, 3:15 PM] thurisaz Emergency Maintenance
We will be upgrading thurisaz (www136) to new hardware. This upgrade will involve
a faster CPU as well as more memory. Downtime should be no more than 10 minutes.
We will post additional information as it becomes available.
[Dec 4, 2001, 12:28 PM] thurisaz Downtime
thurisaz had to be rebooted for the third time this morning. We will be
monitoring the server closely to determine what the causing the server to
crash.
[Dec 4, 2001, 11:18 AM] thurisaz Downtime
thurisaz (www136) had to be rebooted this morning, total downtime was
under 10 minutes.
[Dec 4, 2001, 9:30 AM] thurisaz Downtime
thurisaz (www136) had to be rebooted this morning, downtime was under 15
minutes.
[Dec 3, 2001, 4:08 PM] beith Maintenance Completed
The hard drive swap for beith (www17) has been completed. Total downtime was
approximately 15 minutes.
[Dec 3, 2001, 1:19 PM] Pairlist Signups Issue Resolved
We have located and resolved the problem with Pairlist signups, and all lists
should now be functioning properly. If you are still experiencing problems
with your mailing list, please contact support@pairlist.net. We apologize
for any inconvenience which this has caused.
[Dec 3, 2001, 1:01 PM] dyyme Downtime
dyyme (www71) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Dec 3, 2001, 12:03 PM] Pairlist Signup Problems
An issue has arisen with Pairlist which is preventing new users from
subscribing to mailing lists. This only affects mailing lists that do not
require administrative approval to sign up. At this time, we are working to
locate and resolve this problem. In the meantime, we would recommend that
list administrators set their mailing list subscription settings to require
administrative approval, and to check every few hours to approve new
requests. We apologize for any inconvenience that this may cause.
[Dec 3, 2001, 10:25 AM] mhoh Downtime
mhoh (www191) crashed under heavy load, and had to be manually rebooted.
Total downtime was under 10 minutes.
[Dec 2, 2001, 2:11 PM] beith Emergency Maintenance
We are about to beging an emergency drive swap on beith (www17). There will
be two brief periods of downtime, both about 10 minutes each, at the
beginning and end of this maintenance.
We will post further information as it becomes available.
[Dec 2, 2001, 3:28 AM] beith Emergency Maintenance
We have restored beith (www17) to normal operation on an emergency basis.
We have scheduled an emergency drive swap for within the next 24 hours,
and will be monitoring the server closely for any further problems. No
customer data has been lost during this failure, and data backups are
current and complete. We will post further notices when the swap occurs.
[Dec 2, 2001, 2:23 AM] beith Emergency Maintenance
beith (www17) has had a harddrive failure. We are performing an emergency rebuild.
Services may be unavailable for another 30 minutes.
[Dec 2, 2001, 12:43 AM] ynilo Downtime
ynilo (www166) crashed under heavy load and had to be rebooted. Its harddrive was
manually cleaned, and the server is now operating normally. Total downtime was
approximately 30 minutes.
[Dec 1, 2001, 8:54 PM] iski Downtime
iski (www242) was rebooted to clear up a problematic load condition. Downtime was
approximately one minute.
[Nov 30, 2001, 5:59 AM] zatz Downtime
zatz (www122) crashed under heavy load and had to be rebooted. Downtime
was less than 10 minutes.
[Nov 29, 2001, 1:01 PM] vepar Downtime
vepar (www201) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Nov 27, 2001, 6:56 PM] anca Downtime
anca (www53) failed with hardware problems, and was swapped into a new
chasis (P3-1000). This server has since returned to normal service.
[Nov 27, 2001, 11:02 AM] emancholl Downtime
emancholl (www37) crashed due to high load and was manually rebooted this morning.
Downtime was under 10 minutes.
[Nov 26, 2001, 11:55 AM] vepar Downtime
vepar (www201) crashed under heavy load and had to be manually rebooted.
Total downtime was under 10 minutes.
[Nov 24, 2001, 6:59 AM] mhoh Downtime
mhoh (www191) crashed under heavy load and had to be manually rebooted.
Total downtime was about 5 minutes.
[Nov 24, 2001, 1:02 AM] db19 Maintenance Completed
We have completed the drive swap on db19.pair.com. Total downtime was about
20 minutes.
[Nov 24, 2001, 12:21 AM] db7 Downtime
db7.pair.con crashed under high load and had to be manually rebooted. Total
downtime was about 5 minutes.
[Nov 23, 2001, 11:11 PM] db19 Emergency Maintenance
We have detected errors on the hard drive in db19.pair.com. We have
detected errors in the hard drive on db3.pair.com. We will be taking
down the server on an emergency basis for a hard drive swap. There will be
two brief periods of downtime, each of which should be no longer than 10
minutes. We will post more information as it becomes available.
[Nov 21, 2001, 5:24 PM] db18 Maintenance Completed
The drive maintenance on db18 has been completed, and this server has
returned to normal service.
[Nov 21, 2001, 3:27 PM] db18 Emergency Maintenance
We have detected errors in the hard drive on db18.pair.com. We will be taking
down the server on an emergency basis for a hard drive swap. There will be
two brief periods of downtime, each of which should be no longer than 10
minutes. We will post more information as it becomes available.
[Nov 20, 2001, 10:49 AM] uilen Downtime
uilen (www35) crashed under heavy load, and had to be rebooted. Total
downtime was approximately 10 minutes.
[Nov 16, 2001, 6:20 PM] foma Downtime
foma (www176) crashed under heavy load and had to be rebooted. Downtime was
under 5 minutes.
[Nov 12, 2001, 1:28 PM] db3 Maintenance Completed
The hard drive swap on db3.pair.com has been completed. Total downtime was
approximately 10 minutes.
[Nov 12, 2001, 10:01 AM] db3 Emergency Maintenance
We have detected errors in the hard drive on db3.pair.com. We will be taking down
the server on an emergency basis for a hard drive swap. There will be two brief
periods of downtime, each of which should be no longer than 10 minutes. We will
post more information as it becomes available.
[Nov 11, 2001, 7:41 PM] Network Maintenance
During the next three hours, we will be upgrading some of the cable plant
in our datacenter. As the network topology shifts, some servers may
become unreachable for approximately 15 seconds. We will continue to
monitor all servers closely during these upgrades.
[Nov 11, 2001, 7:32 PM] emlis Downtime
emlis (www216) crashed under heavy load this evening. Extensive manual
work was required to restore the filesystem to normal operation.
[Nov 9, 2001, 2:02 PM] idad Downtime
idad (www32) froze under heavy load, and was rebooted. It has since
returned to normal service.
[Nov 8, 2001, 4:02 PM] beith, tinco Planned Maintenance
This evening, these servers will be taken down briefly to replace faulty
CPU fans.
[Nov 7, 2001, 2:50 PM] orvni Downtime
orvni (www205) crashed under heavy load and had to be rebooted. Total
downtime was approximately 15 minutes.
[Nov 5, 2001, 5:24 PM] bemnet Maintenance Completed
The upgrade of bemnet (www158) has been completed. The server is now an Athlon
1000MHz, with 512MB of RAM. Downtime was approximately five minutes.
[Nov 5, 2001, 12:52 PM] bemnet Planned Maintenance
We are planning to upgrade bemnet (www158) shortly after 5:00PM Eastern today.
This will be an upgrade of both processor speed, and memory. It will greatly
improve performance for all users on this server. Downtime should be under ten
minutes. We will post more information as it becomes available.
[Nov 5, 2001, 11:12 AM] bemnet Downtime
bemnet (www158) crashed under heavy load and had to be manually rebooted.
Total downtime was under 10 minutes.
[Nov 4, 2001, 5:56 PM] emra Downtime
emra(www183) crashed under high load. Total downtime was approximately 20
minutes.
[Nov 3, 2001, 5:23 PM] whury Downtime
whury(www179) crashed and had to be manually rebooted. Total downtime was
under 10 minutes.
[Oct 31, 2001, 2:36 PM] qaske Downtime
qaske (www250) crashed under high load. The downtime was under 10 minutes.
[Oct 30, 2001, 6:25 PM] theta Maintenance Completed
We have completed the drive swap on theta (www4). Total downtime for this
maintenance was under 10 minutes.
[Oct 30, 2001, 9:50 AM] kodh Mail Delivery
Incoming mail delivery on kodh (www90) was disabled overnight as a result
of yesterday's drive rebuild. Deliveries have been re-enabled this
morning, and all pending e-mail should be delivered over the next one to
three hours. We apologize for the inconvenience.
[Oct 29, 2001, 11:08 PM] iruvul Maintenance Completed
The hard drive swap on iruvul (www239) has been completed. Total downtime
was under 10 minutes.
[Oct 29, 2001, 10:08 PM] Network Problems
This evening, we are seeing high latency between our network and many
@Home cable modem subscribers. This is often caused by a network outage
or attack within their network, which leads to overloading of their
available peering points and transit connections. The problem is not
likely to persist, but if it does, we will attempt to reroute the traffic.
This type of problem is usually beyond the control of anyone except @Home
itself, however.
[Oct 29, 2001, 9:13 PM] iruvul Emergency Maintenance
iruvul (www239) is showing errors on its hard drive. We are therefore
performing an emergency drive swap, which will include two brief periods of
downtime.
[Oct 29, 2001, 9:05 PM] kodh Maintenance Completed
kodh (www90) has been restored to normal service. The most recent backup
was completed on Sunday evening, and all files were restored successfully
from backup. If you are missing any files, please contact us at
urgent@pair.com.
We apologize for this disruption and downtime; although such a complete
failure is the exception rather than the rule, we will still work to
improve turnaround time for this failure mode. We appreciate your patience
and understanding.
[Oct 29, 2001, 3:31 PM] theta Emergency Maintenance
We have observed a warning condition on the hard drive in theta (www4)
which may indicate an imminent failure. Consequently, we will be taking
this server down shortly to initiate a drive swap. There will be two brief
periods of downtime, less than ten minutes each. Additional notices will be
posted when the swap is completed.
[Oct 29, 2001, 3:16 PM] kodh Emergency Maintenance
Restoration of customer files is taking longer than projected due to the
extraordinarily large number of files stored in approximately thirty
home directories and fifty Web directories. Although service has been
restored for other accounts, these large accounts are being restored in
parallel, and may take several more hours to complete.
During the restoration process, incoming mail delivery has been suspended,
as home directories are required for correct mail processing in most
cases. Pending mail deliveries are normally retried for up to five days,
so no incoming mail should be lost as a result.
We are working on methods to improve server restoration time in the
future; this case was unusual in that the drive failure was sudden and
disastrous, with no possibility of recovering files directly from the
failed drive.
[Oct 29, 2001, 1:38 PM] kodh Emergency Maintenance
Restoration of customer files is proceeding quickly on kodh (www90);
we expect the work to be complete within the next 60 minutes. Further
information will be posted as it becomes available.
[Oct 29, 2001, 12:13 PM] kodh Emergency Maintenance
The hard drive in kodh (www90) has suffered a complete failure. We are
rebuilding the server from recent backups; full service should be restored
within the next 90 minutes.
[Oct 29, 2001, 11:58 AM] kodh Emergency Maintenance
kodh (www90) has suffered an apparent hard drive failure. We are working with
backups to bring the server back online in as short a time as possible. We will
post additional information as it becomes available.
[Oct 29, 2001, 11:14 AM] qaske Downtime
qaske suffered under heavy load, and became unresponsive. A server reboot
was necessary, during which time we increased the RAM of this system.
Further load balancing will be preformed on this server as necessary to
improve stability.
[Oct 29, 2001, 10:17 AM] iski Downtime
iski (www242) needed to be manually rebooted this morning. Downtime was
about 20 minutes.
[Oct 27, 2001, 9:12 PM] Registry Maintenance
Due to maintenance being performed by Verisign GRS, the central registry
will be down this evening from 9pm to 1am Eastern time. New domain
registrations will not be possible, although all existing registrations
will continue to function.
If you are planning to register a new domain or signup for a new account
with a domain name, please wait until after 1am to do so.
[Oct 26, 2001, 7:06 AM] Mail delivery delay
The mail queue on qaske (www250) received a flood of spam mail. We are
currently cleaning out the spam mail from the mail queue. Mail service
should return to normal in 24 hours.
[Oct 26, 2001, 6:26 AM] Network Maintenance
Genuity will be taking down our DS-3 circuit for up to 60 minutes this
morning, in order to regroom the circuit for improved performance. This
should ultimately result in faster connections for customers and visitors
utilizing that connection.
[Oct 25, 2001, 11:22 PM] xamsa Maintenance Completed
We have completed the drive swap on xamsa (www244). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 11:21 PM] okkod Maintenance Completed
We have completed the drive swap on okkod (www235). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 10:36 PM] patru Maintenance Completed
We have completed the drive swap on patru (www232). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 10:22 PM] muyal Maintenance Completed
We have completed the drive swap on muyal (www236). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 9:06 PM] xebte Maintenance Completed
We have completed the drive swap on xebte (www230). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 8:38 PM] peswar Maintenance Completed
We have completed the drive swap on peswar (www233). Total downtime for
this maintenance was under 10 minutes.
[Oct 25, 2001, 8:29 PM] nwalme Maintenance Completed
We have completed the drive swap on nwalme (www57). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 6:07 PM] silme Downtime
silme (www64) had to be rebooted to clear up a brief network issue.
Downtime was about 10 minutes.
[Oct 25, 2001, 4:26 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
xebte (www230)
nwalme (www57)
peswar (www233)
patru (www232)
okkod (www235)
muyal (www236)
vondu (www241)
xamsa (www244)
petora (www229) had originally been scheduled to be swapped tonight.
However, it received an emergency swap due to failure on October 13, and
does not need to be swapped again.
All of these swaps follow the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
Additional notices will be posted as these swaps are completed.
[Oct 25, 2001, 1:59 AM] uinus Maintenance Completed
We have completed the drive swap on uinus (www228). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 1:22 AM] ungwe Maintenance Completed
We have completed the drive swap on ungwe (www48). Total downtime for this
maintenance was under 10 minutes.
[Oct 25, 2001, 1:20 AM] silme Maintenance Completed
We have completed the drive swap on silme (www64). Total downtime for this
maintenance was under 10 minutes.
[Oct 24, 2001, 11:25 PM] anga Maintenance Completed
We have completed the drive swap on anga (www47). Total downtime for this
maintenance was under 10 minutes.
[Oct 24, 2001, 10:46 PM] cuzea Maintenance Completed
The hard drive swap on cuzea (www94) has been completed. Total downtime was
approximately 10 minutes.
[Oct 24, 2001, 9:59 PM] lepen Maintenance Completed
We have completed the drive swap on lepen (www129). Total downtime for this
maintenance was under 10 minutes.
[Oct 24, 2001, 8:40 PM] parma Maintenance Completed
We have completed the drive swap on parma (www42). Total downtime for this
maintenance was under 10 minutes.
[Oct 24, 2001, 8:22 PM] quesse Maintenance Completed
We have completed the drive swap on quesse (www44). Total downtime for this
maintenance was under 10 minutes.
[Oct 24, 2001, 5:43 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
lepen (www129)
parma (www42)
quesse (www44)
anga (www47)
ungwe (www48)
silme (www64)
uinus (www228)
cuzea (www94)
These swaps follow the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
Additional notices will be posted as these swaps are completed.
[Oct 24, 2001, 3:11 PM] roomen Maintenance Completed
The hard drive swap for roomen (www61) has been completed. Total downtime was
approximately 15 minutes.
[Oct 24, 2001, 2:34 PM] otok Maintenance Completed
The hard drive swap on otok (www131) has been completed.
Additionally, the server was upgraded to an Athlon 1000MHz, with
512MB RAM. Total downtime was approximately 20 minutes.
[Oct 24, 2001, 11:13 AM] shen Maintenance Completed
The hard drive swap on shen (www88) has been completed. Total downtime was
approximately 15 minutes.
[Oct 24, 2001, 10:49 AM] tama Maintenance Completed
The hard drive swap on tama (www70) has been completed. Total downtime was
approximately 15 minutes.
[Oct 24, 2001, 12:53 AM] neled Maintenance Completed
We have completed the drive swap on neled (www127). Total downtime for this
maintenance was under 10 minutes.
[Oct 24, 2001, 12:18 AM] eahta Maintenance Completed
We have completed the drive swap on eahta (www227). Total downtime for this
maintenance was under 10 minutes.
[Oct 23, 2001, 11:20 PM] dhaame Maintenance Completed
We have completed the drive swap on dhaame (www77). Total downtime for this
maintenance was under 10 minutes.
[Oct 23, 2001, 8:30 PM] tenshian Maintenance Completed
We have completed the drive swap on tenshian (www103). Total downtime for
this maintenance was under 10 minutes.
[Oct 23, 2001, 7:29 PM] Network Problems Resolved
Between 6:28pm and 7:25pm Eastern time, a severe distributed
denial-of-service attack was directed towards our network. We have taken
all possible steps to protect our network and ensure continued availability
and performance. For brief periods during this attack, performance may
have been affected for some visitors.
We have information on the source of this attack, and we will be pursuing
legal recourse against the attackers. We are also working to further
enhance our network for improved protection under these circumstances.
Recent upgrades have already made a dramatic difference in this area.
We appreciate our customers' patience while we deal with these challenges.
[Oct 23, 2001, 7:18 PM] hyarmen Emergency Maintenance
hyarmen (www66) was having network connectivity problems. It is currently
recieving maintainence and should be back in working order in no time.
[Oct 23, 2001, 4:31 PM] qaske Downtime
qaske (www250) crashed under high load and had to be rebooted. Total
downtime was less than 10 minutes.
[Oct 23, 2001, 4:27 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
neled (www127)
tenshian (www103)
dhaame (www77)
shen (www88)
otok (www131)
tama (www70)
roomen (www61)
eahta (www227)
These swaps follow the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
Additional notices will be posted as these swaps are completed.
[Oct 23, 2001, 1:41 PM] xebte Downtime
xebte (www230) needed to be manually rebooted this afternoon. Downtime was
less than 10 minutes.
[Oct 23, 2001, 1:25 PM] silme Downtime
Silme (www64) had to be manually rebooted this afternoon. Total downtime
was less than 10 minutes.
[Oct 23, 2001, 11:48 AM] duwo Maintenance Completed
The hard drive swap for duwo (www219) has been completed. Total downtime was
under fifteen minutes.
[Oct 23, 2001, 11:13 AM] wredhor Maintenance Completed
The hard drive swap for wredhor (www76) has been completed. Total downtime was
under fifteen minutes.
[Oct 23, 2001, 2:52 AM] gwind Maintenance Completed
We have completed the drive swap on gwind (www74). Total downtime for this
maintenance was under 10 minutes.
[Oct 23, 2001, 1:24 AM] gamma Maintenance Completed
We have completed the drive swap on gamma (www2). Total downtime for this
maintenance was under 10 minutes.
[Oct 23, 2001, 12:49 AM] edad Maintenance Completed
We have completed the drive swap on edad (www31). Total downtime for this
maintenance was under 10 minutes.
[Oct 22, 2001, 11:11 PM] yuzz Maintenance Completed
We have completed the drive swap on yuzz (www117). Total downtime for this
maintenance was under 10 minutes.
[Oct 22, 2001, 10:27 PM] zais Maintenance Completed
We have completed the scheduled drive swap on zais (www89). Total downtime
for this maintenance was under 10 minutes.
[Oct 22, 2001, 9:27 PM] coll Maintenance Completed
We have completed the drive swap on coll (www22). Total downtime for this
maintenance was under 10 minutes.
[Oct 22, 2001, 9:16 PM] enek Maintenance Completed
We have completed the drive swap on enek (www130). Total downtime for this
maintenance was under 10 minutes.
[Oct 22, 2001, 5:20 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
enek (www130)
coll (www22)
gamma (www2)
edad (www31)
yuzz (www117)
gwind (www74)
wredhor (www76)
zais (www89)
duwo (www219)
These swaps follow the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
Additional notices will be posted as these swaps are completed.
[Oct 22, 2001, 4:32 PM] qaske Downtime
qaske (www250) was rebooted to clear up a high load issue. Downtime was under one
minute.
[Oct 22, 2001, 2:08 PM] theta Maintenance Completed
The drive swap on theta (www4) has been completed. A RAM upgrade has also
been completed, to 512MB.
[Oct 20, 2001, 10:32 PM] Network Problems
We have identified an attempted denial of service attack on our network.
Some customers may have noticed a period of decreased network performance
starting around 9:30 US eastern time. We believe that we have managed to
filter this attack from affecting us any further and will continue to
monitor the situation closely.
[Oct 19, 2001, 7:30 PM] emancholl Downtime
emancholl (www37) crashed under high load and had to be rebooted. Total
downtime was under 10 minutes.
[Oct 19, 2001, 12:06 PM] muin Maintenance Completed
The hard drive swap for muin (www23) has been completed. Total downtime for the
swap was approximately 15 minutes.
[Oct 19, 2001, 11:09 AM] Network Maintenance
We are reconfiguring our Sprint OC-3c circuit this morning; downtime of
less than 30 seconds is expected, during which time traffic will be carried
by our other providers.
We have also observed two brief periods of downtime (less than five
seconds each) on our Cable and Wireless DS-3. We have contacted their NOC
to monitor the circuit; if maintenance is necessary, it will be done in
the evening to minimize disruptions.
[Oct 18, 2001, 11:57 PM] naam Maintenance Completed
The drive swap of naam (www114) has been completed. RAM has been upgraded
to 512MB. Downtime was about 10 minutes.
[Oct 18, 2001, 11:30 PM] aster Maintenance Completed
We have completed the drive swap on aster (www96). Total downtime for this
maintenance was under 10 minutes.
[Oct 18, 2001, 11:17 PM] bellat Maintenance Completed
The drive swap for bellat (www101) has been completed. RAM has also been
upgraded to 512MB. Total downtime was about 10 minutes.
[Oct 18, 2001, 11:04 PM] nallav Maintenance Completed
We have completed the drive swap on nallav (www237). It also recieved a RAM
upgrade, and now has 512 MB of RAM. Total downtime for this maintenance was
under 10 minutes.
[Oct 18, 2001, 9:58 PM] ulwar RAM
ulwar also received a RAM upgrade, and now has 512 MB of RAM.
[Oct 18, 2001, 9:57 PM] ulwar Maintenance Completed
We have completed the drive swap on ulwar (www82). Total downtime for this
maintenance was under 10 minutes.
[Oct 18, 2001, 9:21 PM] laguz Emergency Maintenance
We will be taking laguz (www148) down briefly to upgrade it's chassis to
better handle current load conditions. An additional notice will be posted
once this upgrade is complete.
[Oct 18, 2001, 8:54 PM] eeoth Maintenance Completed
We have completed the drive swap and RAM upgrade on eeoth (www85). It now
has 512MB of RAM. Total downtime for this maintenance was about 10 minutes.
[Oct 18, 2001, 8:52 PM] ur Maintenance Completed
The hard drive swap on ur (www30) has been completed. Total downtime was
under 10 minutes.
[Oct 18, 2001, 8:17 PM] anca Maintenance Completed
We have completed the drive swap and RAM upgrade on anca (www53). It now
has 512MB of RAM. Total downtime for this maintenance was under 10 minutes.
[Oct 18, 2001, 4:38 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
anca (www53)
ulwar (www82)
eeoth (www85)
nallav (www237)
muin (www23)
ur (www30)
fearn (www40)
bellat (www101)
theta (www4)
naam (www114)
aster (www96)
These swaps follow the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
In addition, anca, ulwar, eeoth, nallav, bellat, theta, and naam will all
receive RAM upgrades.
Additional notices will be posted as the swaps are completed.
[Oct 18, 2001, 4:24 PM] Server Upgrades
This evening, the following servers will be taken down briefly for RAM
upgrades:
psi, nuin, calma, cele, vuae, dhabat
Each downtime will be brief, so there should be little disruption of
service.
[Oct 18, 2001, 3:51 PM] pi Maintenance Completed
The hard drive upgrade on pi (www10) was completed. The server was also upgraded
to have 512MB RAM. Total downtime was approximately 15 minutes.
[Oct 18, 2001, 1:59 PM] laguz Downtime
laguz (www148) crashed under heavy load and had to be manually rebooted. Total
downtime was approximately 10 minutes.
[Oct 18, 2001, 5:31 AM] vepar Downtime
vepar (www201) crashed under heavy load and had to be rebooted. Total
downtime was under 10 minutes.
[Oct 18, 2001, 1:32 AM] yekk Maintenance Completed
The hard drive swap on yekk (www124) has been completed. Total downtime was
under 10 minutes.
[Oct 18, 2001, 1:32 AM] thesel Maintenance Completed
We have completed the scheduled drive swap on thesel (www99). Total
downtime for this maintenance was under 10 minutes.
[Oct 18, 2001, 1:12 AM] uilen Downtime
uilen (www35) crashed under heavy load and had to be rebooted. Total
downtime was 15 minutes.
[Oct 17, 2001, 10:02 PM] dair Maintenance Completed
We have completed the scheduled drive swap on dair (www20). Total downtime
for this maintenance was under 10 minutes.
[Oct 17, 2001, 8:44 PM] zortzi Maintenance Completed
We have completed the scheduled drive swap on zortzi (www243). Total
downtime for this maintenance was under 10 minutes.
[Oct 17, 2001, 8:13 PM] dieci Maintenance Completed
We have completed the scheduled drive swap on dieci (www231). Total
downtime for this maintenance was under 10 minutes.
[Oct 17, 2001, 8:12 PM] naih Maintenance Completed
The hard drive swap on naih (www214) has been completed. Total downtime was
under 15 minutes.
[Oct 17, 2001, 4:00 PM] quan Maintenance Completed
The hard drive swap on quan (www120) has been completed. Total downtime was
under 15 minutes.
[Oct 17, 2001, 3:25 PM] akket Maintenance Completed
The hard drive swap on akket (www211) has been completed. Total downtime was
under 15 minutes.
[Oct 17, 2001, 2:34 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
pi (www10)
dair (www20)
dieci (www231)
thesel (www99)
zortzi (www243)
yekk (www124)
naih (www214)
All of these swaps are following the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
Additional notices will be posted as the swaps are completed.
[Oct 17, 2001, 1:19 PM] upsilon Maintenance Completed
The hard drive swap on upsilon (www13) has been completed. Total downtime was
under 15 minutes.
[Oct 17, 2001, 11:22 AM] Delayed Log Delivery
We have determined that several of our servers failed to deliver usage logs to user directories last night.
We are investigating the issue and will be re-generating logs within the next hour.
[Oct 17, 2001, 3:46 AM] emancholl Downtime
emancholl (www37) crashed under heavy load and had to be rebooted. Total
downtime was under 10 minutes.
[Oct 17, 2001, 12:42 AM] omega Maintenance Completed
We have completed the scheduled drive swap on omega (www16). Total
downtime for this maintenance was under 10 minutes.
[Oct 16, 2001, 11:39 PM] reit Maintenance Completed
We have completed the scheduled drive swap on reit (www108). Total downtime
for this maintenance was under 10 minutes.
[Oct 16, 2001, 9:06 PM] umbar Maintenance Completed
We have completed the scheduled drive swap on umbar (www46). Total downtime
for this maintenance was under 10 minutes.
[Oct 16, 2001, 8:45 PM] foma Maintenance Completed
We have completed the scheduled drive swap on foma (www176). It now has a
30Gb hard drive. Total downtime for this maintenance was under 10 minutes.
[Oct 16, 2001, 6:27 PM] Planned Maintenance Revision
Upon further evaluation, we have concluded that the following drives do not
need to be swapped at this time:
diaf (www215)
oynos (www218)
fearn (www40)
[Oct 16, 2001, 5:33 PM] athani Maintenance Completed
We have completed the drive swap on athani (www157). During the completion
of the swap, the power supply in the server apparetnly died. The disk was
immediately swapped to a new chassis, an Athlon 1000MHz with 256Mb of RAM.
Total downtime for this maintenance was about 15 minutes.
[Oct 16, 2001, 5:02 PM] Planned Maintenance
Tonight we will begin drive swaps on the following servers:
omega (www16)
foma (www176)
umbar (www46)
reit (www108)
akket (www211)
upsilon (www13)
diaf (www215)
quan (www120)
All of these swaps are following the schedule previously posted at:
http://www.pair.com/pair/support/notices/driveswaps.html
Additional notices will be posted as the swaps are completed.
[Oct 16, 2001, 2:58 PM] huath Maintenance Completed
huath (www19) received a hard drive upgrade. Also, the chassis was upgraded to an
Athlon 1000 MHz, with 512 MB of RAM. Total downtime was approximately 15 minutes.
[Oct 16, 2001, 1:35 PM] agho Maintenance Completed
agho (www187) received a hard drive upgrade. Total downtime was less than 15
minutes.
[Oct 16, 2001, 2:00 AM] laguz Maintenance Completed
We have completed the scheduled drive swap on laguz (www148). Total
downtime for this maintenance was under 10 minutes.
[Oct 16, 2001, 1:11 AM] oomur Maintenance Completed
We have completed the scheduled drive swap on oomur (www209). Total
downtime for this maintenance was under 10 minutes.
[Oct 16, 2001, 12:53 AM] lambe Maintenance Completed
We have completed the scheduled drive swap on lambe (www63). Total
downtime was under 10 minutes.
[Oct 16, 2001, 12:49 AM] iota Maintenance Completed
The drive swap on iota (www5) has been completed. All services should be
functioning as normal.
[Oct 15, 2001, 10:40 PM] ingwaz Maintenance Completed
We have completed the scheduled drive swap on ingwaz (www149). Total
downtime was under 10 minutes.
[Oct 15, 2001, 10:13 PM] uruz Maintenance Completed
We have completed the scheduled drive swap on uruz (www135). Total downtime
was under 10 minutes.
[Oct 15, 2001, 7:46 PM] iota Planned Maintenance
iota (www5) is scheduled tonight to have its hard drive swapped. We will
begin the process for this shortly. There will be two brief periods of
downtime.
[Oct 15, 2001, 3:35 PM] laguz Downtime
laguz (www148) crashed under a high load and was rebooted. The downtime
was under 15 minutes.
[Oct 13, 2001, 5:18 AM] petora Downtime
Petora has been restored; however the backups were 9 hours old. Logs and
files added between 8pm and 5am EDT are unfortunately lost.
[Oct 13, 2001, 4:13 AM] petora Downtime
Petora is suffering a serious hard drive crash. We are in the process of
getting it back up as quickly as possible.
[Oct 12, 2001, 6:15 PM] thnad Maintenance Completed
We have completed the drive swap of thnad (www121). Total downtime for this
maintenance was about 10 minutes.
[Oct 12, 2001, 4:39 PM] laguz Downtime
laguz (www148) crashed under high load and had to be rebooted. Total
downtime was under 10 minutes.
[Oct 12, 2001, 4:04 PM] thnad Emergency Maintenance
We have observed a warning condition on the hard drive in thnad (www121)
which may indicate an imminent failure. The hard drive in thnad was
scheduled to be swapped next week as a part of a proactive campaign to swap
out suspected bad drive models. We will be taking this server down on an
emergency basis tonight to begin the drive swap. There will be two brief
periods of downtime, each under ten minutes. Additional information will
be posted as it becomes available.
[Oct 11, 2001, 7:19 PM] ynilo Downtime
Due to high load, ynilo (www166) had to be manually rebooted. Downtime for
this was under ten minutes.
[Oct 11, 2001, 4:19 PM] Server Upgrades
As a part of an ongoing project to upgrade and update our server hardware,
the following servers will be taken down briefly this evening for RAM
upgrades:
psi
nuin
calma
cele
vuae
dhabat
Each downtime will be brief, about 5 minutes while work is completed.
[Oct 10, 2001, 4:18 PM] dotya Downtime
dotya (www170) was rebooted in order to clear up a pervasive internal
problem with its FTP service. Downtime was less than two minutes, and
the problem appears to have been eliminated.
[Oct 10, 2001, 9:17 AM] flure Downtime
flure (www75) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[Oct 8, 2001, 2:59 PM] emancholl Downtime
emancholl (www37) crashed under heavy load. The server was rebooted.
Downtime was 10 minutes.
[Oct 7, 2001, 1:35 AM] emancholl Downtime
emancholl (www37) crashed under heavy load and had to b3e rebooted. Total
downtime was about 10 minutes.
[Oct 6, 2001, 8:18 PM] db4 Maintenance Completed
The hard drive swap on db4.pair.com has been completed. Total downtime was
under ten minutes.
[Oct 6, 2001, 6:59 PM] db4 Emergency Maintenance
We have detected errors on the hard drive in db4.pair.com. We will be swapping
the hard drive on an emergency basis. There will be two brief periods of
downtime, which should be no longer than 10 minutes each. We will post more
information as it becomes available.
[Oct 5, 2001, 3:22 PM] beith Downtime
beith (www17) crashed under high load and had to be rebooted. Total
downtime was under 8 minutes.
[Oct 5, 2001, 2:06 AM] tinco Downtime
Tinco was found with hardware problems after the drive swap. We're
swapping it into a new chassis presently.
[Oct 4, 2001, 5:22 PM] Server Upgrades
The following servers will be taken down briefly this evening for hardware
upgrades:
Downtime will be brief, while hardware is upgraded to increase capacity and
replace older parts.
[Oct 4, 2001, 3:05 AM] halla Downtime
halla (www68) crashed under heavy load and had to be rebooted. Total
downtime was about 10 minutes.
[Oct 4, 2001, 12:58 AM] laguz Downtime
laguz(www48) crashed under heavy load and had to be rebooted. Total
downtime was about 10 minutes.
[Oct 2, 2001, 5:06 PM] delsi Downtime
delsi crashed under heavy load, twice in recent time. We have increased
system memory, which should help prevent future outages of this nature.
[Oct 2, 2001, 10:57 AM] beith Maintenance Completed
The chassis swap on beith has been completed and is now running a Pentium
III processor. We will be watching the server closely for any other
problems. We apologize for the extended time the chassis swap took and for
any inconvenience this may have caused the users on beith.
[Oct 2, 2001, 10:13 AM] beith Emergency Maintenance
Emergency maintenance is being performed on the beith server. A chassis
swap is being performed due to a suspected motherboard problem. The server
will be done approximately 15 minutes. Another notice will be posted when
the maintenance is completed.
[Oct 2, 2001, 9:41 AM] beith Downtime
beith (www17) had to be manually rebooted this morning, total downtime was
less than 10 minutes.
[Oct 1, 2001, 11:31 PM] Network Problems Resolved
Sprint is now routing @Home traffic through East Coast peering, and we
are seeing no latency problems on that path. We believe that the earlier
problems were temporary, but we are following up with Sprint to ensure
that we can expect good performance to @Home in the future.
[Oct 1, 2001, 8:31 PM] eahta Downtime
eahta (www227) crashed under high load and had to be rebooted. Total
downtime was less than 20 minutes.
[Oct 1, 2001, 6:11 PM] Network Problems
Sprintlink is currently having peering problems with @Home; cable modem
users of @Home-based services may experience slow throughput to our
network. We are escalating this matter with Sprint, and hope to have
it resolved this evening. If it is not resolved, we will move the traffic
to a different provider.
[Oct 1, 2001, 1:12 PM] db7 Maintenance Completed
The hard drive swap for db7.pair.com has been completed. Total downtime was
approximately 10 minutes.
[Oct 1, 2001, 11:08 AM] db7 Emergency Maintenance
We have detected errors on the hard drive in db7.pair.com. We will be swapping
the hard drive on an emergency basis. There will be two brief periods of
downtime, which should be no longer than 10 minutes each. We will post more
information as it becomes available.
[Oct 1, 2001, 10:30 AM] noldo Maintenance Completed
The hard drive swap for noldo (www56) has been completed. Total downtime was
under ten minutes.
[Oct 1, 2001, 9:29 AM] New Relay Server
Our relay server, relay.pair.com, is being replaced with a much faster
and more efficient server, effective immediately. The new server will
transparently replace the old within the next few minutes. Any prior
authentications made by accessing a mailbox in the past 15 minutes will
not be retained; you must access your mailbox again after the new server
is running. You may continue to use the relay service in the same
fashion as always; there are no changes to functionality. The old server
will flush out its pending mail deliveries with no loss of mail.
We feel that this upgrade will dramatically improve the performance and
reliability of our relay service. We are also working on further
enhancements in that direction.
[Sep 30, 2001, 7:38 PM] noldo Emergency Maintenance
We will be taking down noldo (www56) for an emergency drive swap as its drive has
showed signs of impending failure. There will be two brief periods of downtime,
each of which should be no longer than ten minutes. We will post more information
as it becomes available.
[Sep 30, 2001, 4:52 PM] naam Downtime
naam(www114) crashed under high load and had to be manually rebooted. Total
downtime was approximately 15 minutes.
[Sep 30, 2001, 12:43 AM] Relay Server
Throughout the day, work was done to restore our relay server to normal
operation, after a massive Spam campaign was apparently launched by one
of our customers. During the cleanup, a bug may have caused some queued
mail to be misdelivered. This problem has been rectified. If you have
sent important mail through our relay in the past 36 hours, you may wish
to resend it, or follow up with the recipient to ensure it was received.
We have prepared a replacement server which is more efficient and scalable;
the replacement will be transparently swapped into service on Monday
morning. We do not expect further problems with this service thereafter.
Please accept our apologies for these problems, which we have been working
hard to resolve.
[Sep 29, 2001, 9:28 PM] Domain Registration Maintenance
The central domain registry, operated by Verisign, is down for maintenance
this evening. It is expected that new domain registrations will be
unavailable for the next three to four hours. New registrations should be
possible after 1am Eastern time.
All existing registrations will continue to function normally.
[Sep 29, 2001, 11:03 AM] laguz Downtime
laguz (www148) crashed under high load and had to be rebooted.
Total downtime was approximately 10 minutes.
[Sep 29, 2001, 5:43 AM] Relay Server Delay
A large amount of spam was sent through the relay server which has caused a
tremendous queue backup. We are working on getting the spam mail cleaned
out and the messages sent through as quickly as possible.
[Sep 28, 2001, 11:36 PM] diviy Maintenance Completed
Maintenance begun with last night's upgrade has been completed, with a
drive swap this evening.
[Sep 28, 2001, 6:39 PM] delsi Downtime
delsi (www206) crshed under high load and had to be rebooted. Total
downtime was about 8 minutes.
[Sep 28, 2001, 12:54 PM] db9 Maintenance Completed
The hard drive swap on db9.pair.com has been completed. Total downtime was under
ten minutes.
[Sep 28, 2001, 9:54 AM] db9 Emergency Maintenance
We have detected a failing hard drive in db9.pair.com. We will be swapping the
hard drive on an emergency basis. There will be two periods of downtime, each
approximately 10 minutes or less. We will post more information as it becomes
available.
[Sep 28, 2001, 1:44 AM] pi Downtime
pi(www10) crashed under heavy load and had to be rebooted. Total downtime
was about 15 minutes.
[Sep 27, 2001, 4:19 PM] omicron Maintenance Completed
After another chassis swap, omicron (www9) has been running successfully
for the past 90 minutes. A minor misconfiguration was corrected which
affected a few customer sites.
The failure of the previous motherboard was traced to an errant screw
causing a short in the case. Needless to say, we will watch for such
unusual problems more closely in the future.
We extend our apologies to the customers affected by this server's
downtime, and appreciate your patience during its downtime.
[Sep 27, 2001, 2:37 PM] omicron Emergency Maintenance
Shortly after being brought up after the completed drive swap, the
motherboard in omicron (www9) appears to have failed. We are swapping
the drive into a new chassis, and expect to have it online within the
next fifteen minutes.
[Sep 27, 2001, 2:28 PM] omicron Maintenance Completed
The maintenance on omicron (www9) has been completed. The server was set up with
a new chassis and hard drive. We will continue to monitor the server closely for
any problems, though we fully believe that the server will now be stable and will
perform reliably. Total downtime was approximately 20 minutes.
[Sep 27, 2001, 10:47 AM] Server upgrades
The following servers will be taken down this evening for hardware upgrades:
diviy
ynilo
ngatwo
uruz
Each downtime will be brief, from 5-15 minutes while work is done. This is
a part of a project of on-going upgrades to improve the service and capacity
for our webservers.
[Sep 27, 2001, 12:30 AM] ngatwo Downtime
ngatwo(www115) crashed under heavy load and had to be rebooted. Total
downtime was about 10 minutes.
[Sep 26, 2001, 5:53 AM] omicron Downtime
omicron (www9) has crashed six times in the past thirty minutes. We
suspected a memory problem, and replaced and upped the RAM in the server.
This did not rectify the problem, we will swap the chassis and
motherboard, and lastly, initiate a drive swap. When further steps are
warranted, we will post further information.
[Sep 25, 2001, 5:36 PM] Email Delays
The gamma server(www2) had it's mail queue overloaded with some invalid
mail recently. We are currently cleaning out the invalid mail. Mail
Service should return to normal in 24 to 36 hours.
[Sep 25, 2001, 1:01 PM] xebte Downtime
xebte (www230) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Sep 25, 2001, 12:38 PM] New OC-3c Circuit Activated
Our OC-3c circuit to Sprint has been activated this morning. The existing
DS-3 circuit will remain on standby. This increases the capacity of our
network significantly.
This is part of our ongoing upgrade plans as outlined at:
http://www.pair.com/pair/support/notices/upgrades.html
[Sep 25, 2001, 12:37 PM] New DS-3 Circuit Activated
Our DS-3 circuit to ATT has been activated this morning. This should
improve performance for some customers and visitors, and contributes
overall to the robustness and capacity of our network.
This is part of our ongoing upgrade plans as outlined at:
http://www.pair.com/pair/support/notices/upgrades.html
[Sep 25, 2001, 9:41 AM] New DS-3 Circuit Activated
Our DS-3 circuit to Cable and Wireless in Cleveland has been activated
this morning. This should improve performance for some customers and
visitors, and contributes overall to the robustness and capacity of our
network.
This is part of our ongoing upgrade plans as outlined at:
http://www.pair.com/pair/support/notices/upgrades.html
[Sep 24, 2001, 4:07 PM] theta E-mail Delays
We are currently cleaning the mail queue of theta (www4). A large amount of
invalid mail has been recentlyu placed in the queue. Cleaning the queue may
take 2-4 hours. During this time mail delivery may be deferred for up to 36
hours.
[Sep 24, 2001, 11:30 AM] diviy Downtime
diviy (www217) crashed under heavy load, and had to be rebooted. Total
downtime was approximately 15 minutes.
[Sep 24, 2001, 10:56 AM] pi Downtime
pi (www10) crashed under heavy load and had to be rebooted. Downtime was
under 10 minutes.
[Sep 23, 2001, 8:22 PM] diaf Maintenance Completed
We have completed the drive swap on diaf (www215). Total downtime for this
maintenance was under 10 minutes.
[Sep 23, 2001, 7:36 PM] rendu Downtime
rendu (www240) crashed under high load and had to be rebooted. Total
downtime was about 10 minutes.
[Sep 23, 2001, 2:14 PM] diaf Emergency Maintenance
We have observed a warning condition on the hard drive in diaf (www215)
which may indicate an imminent failure. We will be taking this server down
shortly on an emergency basis to initiate a drive swap. There will be two
brief periods of downtime less than ten minutes each. The server should
remain online at all other times. An additional notice will be posted when
the maintenance is completed.
[Sep 23, 2001, 12:05 AM] emancholl Downtime
emancholl(www37) crashed under heavy load and had to be rebooted. Downtime
was about 10 minutes.
[Sep 22, 2001, 1:28 AM] Server Outage Resolved
Immediately after midnight Eastern time, the power distribution strip
in one of our datacenter racks failed, which disrupted power to an
Extreme switch and fourteen servers. Because of the switch downtime,
servers in adjacent racks were briefly affected.
The switch was promptly moved to a different power source, and we began
to track down the power problem. No failure was found in the power
circuit or wiring, so we replaced the strip. This allowed us to restore
power to each affected server, on a time-delay basis. Four user servers,
specifically wredhor (www76), bellat (www101), tenshian (www103), and
diaf (www215), required manual filesystem cleaning in order to resume
service. In the case of diaf, a hard drive error was found; we will
schedule a maintenance window to replace its drive within the next 72
hours.
We will be resuming the interrupted log generation process for the
affected servers. If you have still not received logs by 7am on Saturday,
please contact us at urgent@pair.com.
Please accept our apologies for this disruption of service. We will be
working to identify any factors which may be related to the failure, and
how we can best address those factors.
[Sep 22, 2001, 12:15 AM] Network Problems
A number of our servers are down due to a fuse blown out on one of our
switches. We're working on bringing the outlet back up presently.
[Sep 21, 2001, 9:32 AM] emancholl Downtime
emancholl(www37) had to be manually rebooted this morning. Total downtime
was less than 5 minutes.
[Sep 20, 2001, 5:16 PM] fearn Maintenance Completed
We have completed the drive swap on fearn (www40). Total downtime for this
maintenance was under 10 minutes.
[Sep 20, 2001, 4:04 PM] nuin Downtime
nuin (www18) crashed while in service. It was rebooted and brought back in
to service. Downtime was 10 minutes.
[Sep 20, 2001, 10:34 AM] dhabat Downtime
dhabat (www105) crashed under heavy load and had to be rebooted. Total
downtime was approximately 15 minutes.
[Sep 20, 2001, 2:52 AM] dhabat Downtime
dhabat(www105) crashed under heavy load and had to be rebooted. Total
downtime was about 5 minutes.
[Sep 19, 2001, 12:51 PM] fearn Planned Maintenance
We have observed a warning condition on the hard drive in fearn (www40)
which may indicate an imminent failure. Consequently, this evening after
5PM Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime less than ten minutes
each. An additional notice will be posted when the maintenance is
completed.
[Sep 19, 2001, 10:47 AM] emancholl Downtime
emancholl (www37) crashed under heavy load and had to be rebooted. Total
downtime was approximately five minutes.
[Sep 18, 2001, 11:13 PM] diaf Maintenance Completed
We have completed the drive swap on diaf (www215). Total downtime for this
maintenance was under 10 minutes.
[Sep 18, 2001, 1:07 PM] diaf Planned Maintenance
We have observed a warning condition on the hard drive in diaf (www215)
which may indicate an imminent failure. Consequently, this evening after
5PM Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime less than ten minutes
each. An additional notice will be posted when the maintenance is completed.
[Sep 18, 2001, 1:01 PM] Update on NT Worm Attack
The new Windows NT-based worm that has been attacking hosts throughout the
Internet today has been named "Nimda" by some researchers. The traffic
load on our network has been sustained at approximately 8000 hits/second.
We have responded with network filtering and RAM upgrades for several
heavily-loaded servers.
Please note that none of our servers are vulnerable to the worm
itself, as we do not use Windows NT to provide any services. More
information on this new worm is available at:
We will continue to monitor this network-wide attack for further
developments.
[Sep 18, 2001, 12:56 PM] ingwaz Downtime
ingwaz (www149) crashed under load and was rebooted. Downtime was less
than 15 minutes.
[Sep 18, 2001, 12:43 PM] Server Upgrades
In response to a sudden increase in network traffic due to the emergence of
a new IIS worm this morning, we will be upgrading several of our servers
for increased RAM or CPU to handle this load. Each outage will be brief,
from 5 to 15 minutes while work is completed.
Please note also that the worm in question does not affect our servers,
only Microsoft IIS web servers. However, the exploit attempts approach the
level of a Denial of Service attack from the number of hosts infected. So,
we are taking protective measures to insure that customer service will not
be unduly interrupted by this event.
[Sep 18, 2001, 11:48 AM] flure, ingwaz Downtime
flure(www75) and ingwaz(www149) had to be manually rebooted this morning,
downtime was under 15 minutes.
[Sep 18, 2001, 11:47 AM] nuin Downtime
nuin(www18) had to be manually rebooted this morning, downtime was around
10 minutes.
[Sep 18, 2001, 11:14 AM] ulwar Downtime
ulwar (www82) crashed under heavy load and had to be rebooted. Total
downtime was approximately 15 minutes.
[Sep 18, 2001, 10:42 AM] beith Downtime
beith(www17) had to be manually rebooted this morning, total downtime was
about 10 minutes.
[Sep 18, 2001, 10:17 AM] jera, ingwaz Downtime
Ingwaz (www149) and jera (142) had to be manually rebooted this morning.
Total downtime for each server was under 15 minutes.
[Sep 18, 2001, 9:58 AM] Network Problems
Beginning around 9:20am this morning, a terrific volume of Windows NT-based
probes began to hit our network. These probes are randomly generated by
Windows NT servers on the Internet that have been infected by the Code Red
worm or other variations. Although these probes present no threat to our
FreeBSD servers, the volume of traffic is reducing performance for some
servers.
We are working to mitigate the effect on servers, as well as to restore
service to any server that crashes under load. We are currently receiving
over 8000 hits per second from as many as one hundred thousand NT servers
on the Internet. Network performance has not been affected.
[Sep 18, 2001, 5:01 AM] emancholl Downtime
emancholl(www37) crashed under heavy load and had to be rebooted. Downtime
was less then 10 minutes.
[Sep 17, 2001, 11:35 AM] gebo Downtime
gebo (www139) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Sep 17, 2001, 3:33 AM] voot Downtime
voot crashed under high load. Downtime was about 10 minutes.
[Sep 15, 2001, 5:03 PM] laguz Downtime
laguz(www148) crashed and had to be manually rebooted. Total downtime was
approximately 10 minutes.
[Sep 13, 2001, 9:53 PM] lambe Maintenance Completed
We have completed the drive swap on lambe (www63). Total downtime for this
maintenance was under 10 minutes.
[Sep 13, 2001, 8:25 PM] oynos Maintenance Completed
We have completed the drive swap on oynos (www218). Total downtime for
this maintenance was under 10 minutes.
[Sep 13, 2001, 10:57 AM] lambe, oynos Planned Maintenance
We have observed a warning condition on the hard drives in lambe (www63)
and oynos (www218) which may indicate an imminent failure. Consequently,
this evening after 5PM Eastern time, we will be taking the servers offline
to initiate drive swaps. There will be two brief periods of downtime for
each server, less than ten minutes each. An additional notice will be
posted when the maintenance is completed.
[Sep 13, 2001, 7:38 AM] emancholl Downtime
emancholl(www37) crashed under high load and had to be rebooted. Total
downtime was about 5 minutes.
[Sep 12, 2001, 9:05 PM] vepar Downtime
vepar (www201) failed under heavy load. It has been rebooted and is now
back in service. Downtime was under 5 minutes.
[Sep 11, 2001, 9:24 PM] paell Downtime
paell (www153) was down this evening for approximately 30 minutes. It has
since returned to normal operation.
[Sep 10, 2001, 5:58 PM] ns0 Maintenance Completed
The upgrade of ns0.ns0.com has been completed. Downtime was approximately ten
minutes.
[Sep 10, 2001, 1:32 PM] ns0.ns0.com Planned Maintenance
We are planning to upgrade ns0.ns0.com today, Sept. 10, at approximately 5:00 PM
Eastern. Downtime should be ten minutes or less. We will post more information
as it becomes available.
[Sep 10, 2001, 10:50 AM] cele Downtime
cele (www73) was rebooted to clear up a persistent high load condition. Downtime
was under one minute.
[Sep 8, 2001, 5:31 PM] ceirt Downtime
ceirt(www39) crashed and had to be manually rebooted. Totally downtime was
under 10 minutes.
[Sep 7, 2001, 1:35 PM] pi Downtime
pi (www10) crashed under high load and had to be rebooted. Total downtime
was about 5 minutes.
[Sep 6, 2001, 11:06 AM] pi Downtime
pi (www10) crashed under high load and had to be rebooted. Total downtime
for this was about 10 minutes.
[Sep 6, 2001, 3:33 AM] voot Downtime
voot(www154) crashed under heavy load and had to be rebooted. Downtime was
less then 10
[Sep 5, 2001, 1:24 PM] aster Maintenance Completed
We have completed the drive swap on aster (www96). Total downtime for this
maintenance was under 10 minutes.
[Sep 5, 2001, 10:28 AM] neter Downtime
neter (www132) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Sep 4, 2001, 5:38 PM] agho Maintenance Completed
We have completed the RAM upgrade of agho (www187). It now has 384Mb of
RAM. Total downtime for this upgrade was about 15 minutes.
[Sep 4, 2001, 3:41 PM] agho Planned Maintenance
We will be taking agho (www187) down breifly this evening to add more RAM
to it. An additional notice will be posted once this upgrade is completed.
[Sep 4, 2001, 1:07 PM] aster Planned Maintenance
We have observed a warning condition on the hard drive in aster (www96)
which may indicate an imminent failure. Consequently, this evening after
5PM Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime, less than ten minutes
each. An additional notice will be posted when the maintenance is
completed.
[Sep 1, 2001, 10:09 PM] omicron Downtime
omicron (www9) crashed under heavy load and was rebooted. The downtime was
under 10 minutes.
[Aug 31, 2001, 3:28 PM] vert Maintenance Completed
The hard drive swap on vert (www102) has been completed. Total downtime was
approximately 10 minutes.
[Aug 31, 2001, 10:35 AM] dhabat Downtime
dhabat (www105) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Aug 31, 2001, 8:12 AM] emancholl Downtime
emancholl (www37) had to be manually rebooted this morning. Total down time
was less than 10 minutes.
[Aug 31, 2001, 6:52 AM] vert Emergency Maintenance
We have temporarily restored vert (www102) to normal operation. It will
be taken down briefly this morning to begin an emergency drive swap. We
will post further information when that swap is completed.
[Aug 31, 2001, 6:19 AM] vert Emergency Maintenance
vert (www102) crashed with a hard drive read errors. We are currently working to
bring the server back up.
[Aug 30, 2001, 6:35 PM] emlis Maintenance Completed
We have completed the drive swap on emlis (www216). Total downtime for this
maintenance was under 10 minutes.
[Aug 30, 2001, 4:42 PM] Network Maintenance Completed
At our urging, Global Crossing technicians have discovered and rectified
a configuration problem with our new DS-3 circuit. Throughput was being
limited by a configuration error on their end of the circuit, but
performance has now returned to acceptable levels. We apologize to any
customers or site visitors who were affected during the past 90 minutes.
Global Crossing is currently carrying less than 5% of our outbound
traffic.
We will continue to monitor this traffic, as well as tune routing, to
ensure good performance.
[Aug 30, 2001, 3:26 PM] Network Maintenance
Our DS-3 connectivity to Global Crossing was brought online today
around 3pm Eastern time, and is now operating normally. This upgrade
is part of our ongoing network enhancements as documented at:
http://www.pair.com/pair/support/notices/upgrades.html
Some site visitors may have temporarily experienced slower connections
to our network during the turn-up; we worked with Global Crossing to keep
this effect to a minimum.
We are excited about this upgrade, and hope to continue by turning up
additional connectivity to Sprint, ATT, and Cable and Wireless within the
next few days.
[Aug 29, 2001, 1:00 PM] emlis Planned Maintenance
We have observed a warning condition on the hard drive in emlis (www216)
which may indicate an imminent failure. Consequently, this evening after
5PM Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime, less than ten minutes
each. An additional notice will be posted when the maintenance is
completed.
[Aug 28, 2001, 10:59 AM] diviy Downtime
diviy (www217) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[Aug 28, 2001, 10:18 AM] erraj Downtime
erraj (www213) crashed twice under heavy load. After the second crash, we
were able to find the source of the problem and remove it. The server is now
running smoothly again. Total downtime was approximately 25 minutes.
[Aug 26, 2001, 10:17 PM] unque Maintenance Completed
We have completed the drive swap on unque (www54). Total downtime for this
maintenance was about 5 minutes.
[Aug 26, 2001, 1:21 PM] unque Planned Maintenance
We have observed a warning condition on the hard drive in unque (www54)
which may indicate an imminent failure. Consequently, this evening after
5PM Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime, less than ten minutes
each. An additional notice will be posted when the maintenance is
completed.
[Aug 25, 2001, 2:21 AM] laguz Downtime
laguz (www148) crashed under heavy load and had to be rebooted. Total
downtime was under 10 minutes.
[Aug 24, 2001, 12:49 PM] omicron Downtime
omicron (www9) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Aug 24, 2001, 12:50 AM] naam Downtime
naam(www114) crashed under heavy load and had to be rebooted. Total
downtime was under 10 minutes.
[Aug 23, 2001, 9:27 PM] db15 Emergency Maintenance Completion
The emergency drive swap for db15 has been completed. Due to the nature of
the disk corruption, the mysql server was unavailable for the duration of
the swap. Also as a result of the bad hard drive, the main database has be
replace with a copy from backup. Password that have recently been changed
may have defaulted back to a previous version. All user database
information was successfully copied to the new drive.
[Aug 23, 2001, 8:02 PM] db15 Emergency Maintenance
We have detected hard drive errors on db15.pair.com. We will be performing a
hard drive swap for this server on an emergency basis. There will be two
periods of brief downtime, each of which should be no longer than 10 minutes.
We will post additional information as it becomes available.
[Aug 22, 2001, 2:09 PM] Network Maintenance
Please read the updated article on Network Upgrades, as posted at the
top of http://www.pair.com/pair/support/notices/
We are planning to begin major network upgrades on Thursday, August 23,
and for up to a week thereafter. No disruption of customer traffic is
expected, but the upgrades are important and exciting news nonetheless.
[Aug 22, 2001, 11:30 AM] ingwaz Downtime
ingwaz (www149) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Aug 21, 2001, 2:15 PM] db.pair.com Maintenance Completed
We have completed the hard drive swap on db.pair.com. Total downtime was
approximately 10 minutes.
[Aug 21, 2001, 1:34 PM] vepar Downtime
vepar (www201) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Aug 21, 2001, 11:27 AM] db.pair.com Emergency Maintenance
We have detected hard drive errors on db.pair.com. We will be performing a hard
drive swap for this server on an emergency basis. There will be two periods of
brief downtime, each of which should be no longer than 10 minutes. We will post
additional information as it becomes available.
[Aug 21, 2001, 2:55 AM] naam Downtime
naam(www114) crashed under heavy load and had to be rebooted. Downtime was
less then 10 minutes.
[Aug 20, 2001, 4:15 PM] vepar Downtime
vepar (www201) crashed under high load and was rebooted. The downtime was
under 10 minutes.
[Aug 20, 2001, 2:52 PM] vepar Downtime
vepar (www201) was unresponsive due to high load, and had to be reset.
Because the load prevented it from being rebooted cleanly, the server
had to perform a filesystem check. Total downtime was approximately
15 minutes.
[Aug 17, 2001, 6:59 PM] or Maintenance Completed
We have completed the drive swap on or (www34). Total downtime for this
maintenance was about 5 minutes.
[Aug 16, 2001, 10:35 PM] ebad Maintenance Completed
We have completed the drive swap on ebad (www33). Total downtime for this
maintenance was about 5 minutes.
[Aug 16, 2001, 4:27 PM] diviy Downtime
diviy (www217) became unresponsive due to high load, and was rebooted. It
has since returned to service after a file systems check.
[Aug 16, 2001, 12:03 PM] or, ebad Emergency Maintenance
We have observed warning conditions on the hard drives in or (www34) and
ebad (www33) which may indicate an imminent failure. Consequently, this
evening after 5PM Eastern time, we will be taking them offline to initiate
drive swaps. There will be two brief periods of downtime, less than ten
minutes each. Additional notices will be posted once the maintenance is
completed.
[Aug 14, 2001, 10:37 AM] cele downtime
Cele (www73) had to be manually rebooted. Total downtime was less than 15
minutes.
[Aug 12, 2001, 9:08 PM] Domain Name Problems
A problem was unearthed today with the domain names of some customers, and
as a result a few sites were not responding correctly for a short time this
afternoon. We have tracked down the cause of this problem, and are
correcting this for the affected customers.
We are currently adjusting our nameservers so that this will not recur.
In the meantime, if any customer sees a similar problem with their website
not responding correctly, please write to urgent@pair.com and we will take
care of it immediately.
[Aug 12, 2001, 5:08 PM] delsi Downtime
Delsi crashed under heavy load and was manually rebooted. Total downtime
was under 10 minutes.
[Aug 8, 2001, 4:59 PM] diviy Downtime
diviy (www217) was rebooted when it because unresponsive from high load. It
has since returned to normal service.
[Aug 6, 2001, 6:26 PM] shema Maintenance Completed
We have completed the maintenance on shema (www203). It now has 512MB of
RAM. Total downtime for this maintenance was about 5 minutes.
[Aug 6, 2001, 6:10 PM] bhoth Maintenance Completed
We have completed the maintenance on bhoth (www163). It now has 384MB of
RAM. Total downtime for this mainentance was about 5 minutes.
[Aug 6, 2001, 6:09 PM] fehu Maintenance Completed
We have completed the maintenance on fehu (www134). it now has 256MB of
RAM. Downtime for this mainentance was about 5 minutes.
[Aug 6, 2001, 1:20 PM] pairlist Subscriptions
The subscription function for all lists hosted by our pairlist.net service
has been malfunctioning today. This problem was tracked down and
eliminated a short while ago. We apologize for the inconvenience.
[Aug 6, 2001, 11:55 AM] arda Maintenance Completed
The drive swap for arda has been completed. Please report any problems to
support@pair.com.
[Aug 6, 2001, 11:22 AM] fehu, bhoth, shema Planned Maintenance
We will be taking fehu (www134), bhoth (www163) and shema (www203) down
briefly this evening after 6PM EST to add RAM to them. This maintenance
should not take longer than ten minutes per server, and the additional RAM
should help to increase performance. An additional notice will be posted
once the maintenance is completed.
[Aug 5, 2001, 7:34 PM] arda Emergency Maintenance
arda (www62) will be going down momentarily as we perform an emergency hard
drive swap. There will be two periods of downtime lasting approximately 5 minutes each.
[Aug 3, 2001, 5:08 PM] gao Maintenance Completed
We have completed the maintenance on gao (www111). It now has 512MB of RAM.
Total downtime for this maintenance was about 5 minutes.
[Aug 3, 2001, 4:08 PM] gao Planned Maintenance
We will be taking gao (www111) down briefly this evening after 5PM to
increase the amount of RAM in it. An additional notice will be posted
once the maintenance is complete.
[Aug 2, 2001, 6:18 PM] gao Maintenance Completed
We have completed the chassis swap of gao (www111). It is now a Pentium III
1000MHz machine with 256MB of RAM. Total downtime for this maintenance was
about 10 minutes.
[Aug 2, 2001, 9:21 AM] omicron Downtime
Omicron (www9) crashed under heavy load this morning
around 9:00am and had to be manually rebooted. Total
downtime was less than 10 minutes.
[Aug 2, 2001, 12:52 AM] gao Planned Maintenance
The hard drive swap on gao (www111) was completed this evening. However, in
the process we discovered that another bit of maintenance will need to be
done. This will likely be performed Thursday evening, and a notice about it
will be posted when it occurs.
[Aug 1, 2001, 1:07 AM] xrra Maintenance Completed
We have completed the maintenance on xrra (www116). We believe this should
clear up all corruption errors seen on this server recently. The server is
again responding at it's normal operating condition. We apologize for any
inconvenience this maintenance has caused.
[Jul 31, 2001, 9:27 PM] xrra Continuing Maintenance
Unfortunately, the corruption on the primary drive of xrra (www116) is
worse than originally thought. The server crashed again, and the corruption
is such that the primary drive will no longer boot. Due to this, we can no
longer run the normal drive swap proceedure, and will continue "by hand."
This will, unfortunately, result in some downtime for customer sites and
accounts. We will post additional notices as the maintenance continues. We
apologize for the inconvenience.
[Jul 31, 2001, 8:33 PM] gao Emergency Maintenance
gao (www111) will be going down momentarily as we perform an emergency hard
drive swap. There will be two periods of downtime lasting approximately 5
minutes each.
[Jul 31, 2001, 7:22 PM] xrra Continued Maintenance
Due to massive corruption on the primary drive on xrra (www116), we are
experiencing intermittent downtime. We are continuing with the drive swap
which began at 5PM EST; however, certain services may be limited during
this maintenance. We are doing all we can to keep the server operational
during this maintenance, and to limit downtime as much as possible. We
apologize for any inconvenience this causes.
Additional notices will be posted as the maintenance progresses.
[Jul 31, 2001, 2:04 PM] xrra Planned Maintenance
In spite of repeated maintenance performed on xrra to track down and
eliminate its stability problems, we are continuing to see problems in
directory structures on the main drive. These problems are causing
system crashes. We believe these problems arose before the previous
motherboard swap, but in order to eliminate them, we will perform another
drive swap. This will take place after 5pm this evening, with minimal
downtime. After this swap is complete, every component of the system will
have been replaced at least once. We appreciate our customers' patience
while we work to bring this server up to the level of reliability our
service is known for.
[Jul 31, 2001, 8:39 AM] omicron Downtime
Omicron (www9) crashed under heavy load this morning around 8:28AM and had
to be manually rebooted. Total downtime was less than 5 minutes.
[Jul 30, 2001, 9:30 PM] sowilu Downtime
We were able to salvage the primary hard drive on sowilu (www145), and it
now back up. The server has been restored to it's normal working condition.
The drive swap is continuing as scheduled. We apologize for the
inconvenience this added downtime has caused.
[Jul 30, 2001, 8:46 PM] sowilu Downtime
During its emergency drive swap, the current drive on sowilu (www145) has
completely failed. This will unfortunately extend the downtime for this
procedure.
We estimate it will be approximately two to three hours before the server
has been restored to its normal working condition. We will post updates as
they became available, and apologize in the meantime for this inconvenience.
[Jul 30, 2001, 5:59 PM] xrra Downtime
xrra (www116) crashed and had to be rebooted with a manual system check.
Total downtime was about 10 minutes.
[Jul 30, 2001, 5:19 PM] xrra Maintenance Completed
We have completed the chassis swap of xrra (www116). It is now a Pentium
III 1GHz with 256MB of RAM. Total downtime for this maintenance was about 5
minutes.
[Jul 30, 2001, 1:47 PM] xrra Planned Maintenance
Due to a suspected faulty motherboard, we will be taking xrra (www116)
offline at approximately 5:00 PM Eastern, for a full chassis swap. Total
downtime for this maintenance should be about 10 minutes, and an additional
notice will be posted when the maintenance is completed.
[Jul 30, 2001, 1:43 PM] enda Planned Maintenance
Due to size constraints on the hard drive, we will begin a drive upgrade of
enda (www80) tonight. Maintenance will begin after 6PM EST and could last
several hours. There will be two brief periods of downtime, less than ten
minuteseach. An additional notice will be posted once the maintenance is
completed.
[Jul 30, 2001, 1:39 PM] sowilu Planned Maintenance
We have observed a warning condition on the hard drive in sowilu (www145)
which may indicate an imminent failure. Consequently, this evening after
6pm Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime, less than ten minutes
each. An additional notice will be posted once the maintenance is
completed.
[Jul 30, 2001, 1:18 PM] cgiwrap Update
For security reasons, the cgiwrap utility was updated on all user servers
earlier today. The update should not affect any functionality for users.
The php-cgiwrap utility was briefly disrupted; we are working to upgrade
this version of cgiwrap separately and without further disruption.
[Jul 30, 2001, 9:24 AM] xrra Downtime
xrra (www116) crashed, and required a manual filesystem check prior to
returning to service. Downtime was approximately 20 minutes.
[Jul 28, 2001, 9:19 PM] Verisign Registry Maintenance
The central domain registry at Verisign will be down beginning at
9pm Eastern time tonight, for no more than four hours. During this time,
new domain registrations, as well as domain modifications, will not be
available through any of our online systems. We apologize for the belated
notice of this downtime.
Existing domains will continue to work normally.
[Jul 27, 2001, 4:24 PM] xrra Emergency Maintenance
We are planning to take down xrra (www116) at approximately 5:00 PM Eastern, for
emergency maintenance. We will be swapping the server's RAM as it is suspected to
be faulty. Downtime should be no more than 10 minutes. We will post additional
notices as more information becomes available.
[Jul 26, 2001, 12:19 PM] omicron Downtime
omicron (www9) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Jul 25, 2001, 6:48 PM] ebad Downtime
ebad (www33) crashed under high load and had to be rebooted. Total downtime
of 11 minutes.
[Jul 24, 2001, 10:10 PM] dyyme Maintenance Completed
We have completed the drive swap on dyyme (www71). Total downtime for this
maintenance was about 10 minutes.
[Jul 24, 2001, 9:32 PM] ifin Maintenance Completed
We have completed the drive swap on ifin (www36). Total downtime for this
maintenance was about 10 minutes.
[Jul 24, 2001, 12:45 PM] ifin, dyyme Planned Maintenance
We have observed a warning condition on the hard drives in ifin (www36) and
dyyme (www71) which may indicate an imminent failure. Consequently, this
evening after 5pm Eastern time, we will be taking these servers offline to
initiate drive swaps. There will be two brief periods of downtime, less than
ten minutes each. Additional notices will be posted once the maintenance
has been completed.
[Jul 24, 2001, 12:52 AM] bemnet Downtime
bemnet(www158) crashed under heavy load and had to be rebooted. Total
downtime was less then 20 minutes.
[Jul 23, 2001, 9:12 PM] tinne Maintenance Completed
We have completed the drive swap on tinne (www21). Total downtime for this
maintenance was about 5 minutes.
[Jul 23, 2001, 8:39 PM] ruis Maintenance Completed
We have completed the drive swap on ruis (www27). Total downtime for this
maintenance was about 5 minutes.
[Jul 23, 2001, 11:43 AM] tinne, ruis Planned Maintenance
We have observed a warning condition on the hard drives in tinne (www21)
and ruis (www27) which may indicate an imminent failure. Consequently,
this evening after 5PM Eastern time, we will be taking these servers
offline to initiate a drive swap. There will be two brief periods of
downtime, less than ten minutes each. Additional notices will be posted
once the drive swaps have been completed.
[Jul 22, 2001, 4:31 PM] ebad Downtime
ebad (www33) crashed and had to be manually rebooted. Total downtime was under
10 minutes.
[Jul 20, 2001, 9:33 AM] xrra Downtime
xrra (www166) crashed, and required an extensive filesystem check prior to
returning to service. Downtime was approximately 20 minutes.
[Jul 19, 2001, 1:09 PM] Network Problems
Please be advised that the train derailment and resulting chemical spill
in Baltimore, Maryland has disrupted the circuits of a significant number of
Internet carriers, and shifting traffic patterns, as a result, have in some
cases led to reduced performance for many sites. We have received a number
of inquiries about this. Please understand that there is little we can do
on our end while repair work is pending. Carriers have not been allowed
onto the site to begin repairs yet, as far as we know.
[Jul 19, 2001, 9:35 AM] anca Downtime
anca (www53) had to manually run file system check and reboot
total downtime approx. 15 minutes.
[Jul 17, 2001, 10:17 AM] beith Downtime
beith server (www17) had to be manually rebooted this morning, total
downtime was less then 5 minutes.
[Jul 17, 2001, 1:35 AM] xrra Maintenance Completed
The hard drive on xrra (www116) has been successfully swapped. All services
should be functioning as normal.
[Jul 16, 2001, 7:24 PM] xrra Emergency Maintenance
xrra (www116) will be taken down momentarily to perform an emergency hard
drive swap. There will be two brief periods of downtime, each of which
should be no longer than 10 minutes. We will post additional information
once the maintenance has been completed.
[Jul 16, 2001, 4:59 PM] hwesta Maintenance Completed
The hard drive swap on hewsta (www51) has been completed. All services should
be functioning as normal.
[Jul 16, 2001, 12:18 PM] hwesta Emergency Maintenance
hwesta (www51) will be taken down momentarily to perform an emergency hard
drive swap. There will be two brief periods of downtime, each of which
should be no longer than 10 minutes. We will post additional
information once the maintenance is completed.
[Jul 15, 2001, 9:14 AM] muin Maintenance Completed
The hard drive swap on muin (www23) has been completed. All services should
be functioning as normal.
[Jul 15, 2001, 4:49 AM] ur Emergency Maintenance
At this time, we have completed the restoration of files on ur (www30). We
were able to salvage most users' web directories from the failed drive, as
well as most mail spools. All other files were restored from the latest
backups.
[Jul 15, 2001, 3:21 AM] muin Emergency Maintenance
muin (www23) will be taken down momentarily to perform an emergency hard
drive swap. There will be two brief periods of downtime, each of which should
be no longer than 10 minutes. We will post additional information once the
maintenance is completed.
[Jul 15, 2001, 2:00 AM] ur Emergency Maintenance
We are continuing recovery work on ur (www30) after its failure.
At this time, backups are still being uncompressed, and as web directories
are restored, user web sites will come back online. We are also working
to recover whatever we can from the failed drive -- so far we have salvaged
the /var partition, so all main account mailboxes will be current from the
time of the crash.
We would like to correct a point made in the last notice. The last daily
backup was done at approximately 2 PM (EST) on Saturday, July 14. Once
backups are completely deployed, the server will be current as of about
12 hours before the crash. We are still attempting to salvage the /usr
partition of the failed drive, to recover files modified since the last
backup.
[Jul 15, 2001, 12:50 AM] ur Emergency Maintenance
ur (www30) is currently undergoing a full systems recovery from our backup
systems. The old drive in this server, which ironicly was being replaced
by today's maintenance, unrecoverably failed. Our most recent incremental
backup was run on this server just this morning, so loss of
user data will be minimal. Sites hosted on ur may be inaccessable for up
to another hour while files are being recovered. We will post another
notice when this process has been completed.
[Jul 13, 2001, 12:08 PM] ebad Downtime
ebad (www33) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Jul 12, 2001, 9:42 PM] maor Maintenance Completed
The hard drive swap on maor (www113) has been completed. All services
should be functioning as normal.
[Jul 12, 2001, 7:12 PM] maor Emergency Maintenance
maor (www113) will be taken down momentarily to perform an emergency hard
drive swap. There will be two brief periods of downtime, each of which
should be no longer than 10 minutes. We will post additional information
once the maintenance is completed.
[Jul 12, 2001, 12:52 PM] ebad Downtime
ebad (www33) crashed under heavy load, and was brought back online. Downtime was
approximately five minutes.
[Jul 9, 2001, 11:35 PM] vepar Downtime
vepar crashed under heavy load and had to be rebooted. Total downtime was
less then 10 minutes.
[Jul 8, 2001, 8:24 PM] csor Downtime
csor (www190) crashed under heavy load and needed to be rebooted. Downtime
was 10 minutes.
[Jul 5, 2001, 2:12 PM] diviy Downtime
diviy (www217) crashed under heavy load. The downtime was under 20
minutes.
[Jul 5, 2001, 1:20 PM] omicron Downtime
omicron (www9) crashed under heavy load. The downtime was under 10
mintues.
[Jul 5, 2001, 11:49 AM] omicron Downtime
omicron (www9) crashed under high load, and was brought back online. Downtime was
under 10 minutes.
[Jun 29, 2001, 11:25 AM] cele Downtime
cele (www73) ceased responding, and was rebooted. After a filesystem check,
it has returned to normal service.
[Jun 28, 2001, 2:31 PM] agho Downtime
agho (www187) crashed under heavy load. The downtime was under 15
minutes.
[Jun 26, 2001, 3:04 PM] vepar Downtime
vepar (www201) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Jun 26, 2001, 1:51 AM] mini Maintenance Completed
We have completed the drive swap of mini (www126). Total downtime for
this maintenance was about 10 minutes.
[Jun 25, 2001, 11:14 PM] penkwe Maintenance Completed
We have completed the drive swap of penkwe (www220). Total downtime for
this maintenance was about 10 minutes.
[Jun 25, 2001, 12:49 PM] mini, penkwe Planned Maintenance
We have observed a warning condition on the hard drives in both mini
(www126) and penkwe (www220) which may indicate an imminent failure.
Consequently, this evening after 6pm Eastern time, we will be taking
them offline to initiate drive swaps. There will be two brief periods of
downtime, less than ten minutes each.
An additional notice will be posted when each drive swap has been
completed.
[Jun 25, 2001, 9:38 AM] Network Problems Resolved
Despite their denial of any problems, traffic on the Digex circuit returned
to normal levels around 7am Eastern time. We have observed no further
problems since that time.
[Jun 25, 2001, 6:13 AM] Network Problems
Beginning around 5am Eastern time, Digex has apparently had routing
problems in Pittsburgh; traffic on their circuit has dropped to nearly zero
for most of the hour, although they claim there is no problem. Because
their router is accepting and propagating our BGP routes, but blackholing
the traffic, the affected traffic is not being carried by alternate
providers.
We are working with Digex to resolve the matter promptly; if necessary, we
will turn down the BGP session in order to move traffic to alternate paths.
[Jun 23, 2001, 7:35 PM] mildh Downtime
mildh (www87) crashed under load. It was rebooted and is now back in
service. Approximate downtime was 10 minutes.
[Jun 22, 2001, 1:52 PM] Network Problems Resolved
Service has been restored for our UUnet OC-3. All circuits are functioning
normally again.
[Jun 22, 2001, 1:39 PM] Network Problems
All circuits have been restored except for our UUNET circuit. We are
currently working to get this back up and running, but in the mean time,
all traffic should be handled by our other connections. You may notice
a slight degradation in performance until this is restored. We will
provide more information as it becomes available.
[Jun 22, 2001, 1:27 PM] Network Problems
While Verizon has been working on our phone system, a problem has resulted
in several of our lines being down. We are investigating the problem
now, and will post details when more are available.
[Jun 22, 2001, 9:40 AM] omicron Downtime
omicron (www9) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Jun 21, 2001, 6:29 PM] voot Downtime
voot (www154) crashed under high load and had to be rebooted. Total
downtime was less than 10 minutes.
[Jun 21, 2001, 4:39 PM] ampa Maintenance Completed
The hard drive swap for ampa (www52) has been completed. Total downtime
for both periods was under 10 minutes.
[Jun 21, 2001, 4:29 PM] diviy Downtime
diviy (www217) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Jun 21, 2001, 11:23 AM] ampa Emergency Maintenance
We will be taking down ampa (www52) for emergency maintance. The server's
hard drive has developed problems. There will be two brief periods of
downtime, each of which should be no longer than ten minutes. We will post
additional information as it becomes available.
[Jun 18, 2001, 1:06 PM] fearn Downtime
fearn (www40) crashed under high load and had to be rebooted. Total
downtime was about 15 minutes.
[Jun 18, 2001, 9:59 AM] Network Problems Resolved
Our Sprint line has returned to service, and is carrying traffic normally for
us again. We are still awaiting a full explanation for the extended outage
following their scheduled maintenance.
[Jun 18, 2001, 9:16 AM] Network Problems
Sprint's maintenance window has been extended to 7am. However, we still
have not resumed service from them at this time. We are currently
investigating the problem with Sprint, and will post additional details
once we have an explanation for this extended outage.
[Jun 18, 2001, 2:37 AM] Network Maintenance
Sprint has scheduled a major maintenance window for this morning, which
has begun at 2:30am Eastern time. The duration will be as long as four
hours, but customer traffic is flowing easily through alternate providers.
If there is any extension beyond four hours, we will post further
information.
[Jun 15, 2001, 7:32 PM] yanta Maintenance Completed
We have completed the drive swap on yanta (www67). Total downtime for this
maintenance was about 10 minutes.
[Jun 15, 2001, 4:52 PM] fearn Downtime
fearn (www40) crashed under high load and had to be rebooted. Total
downtime was about 10 minutes.
[Jun 15, 2001, 2:51 PM] yanta Emergency Maintenance
We have observed a warning condition on the hard drive in yanta (www67)
which may indicate an imminent failure. We will be taking yanta down
shortly to initiate an emergency drive swap. There will be two brief
periods of downtime, less than ten minutes each. We will post an additional
notice when the swap has been completed.
[Jun 14, 2001, 3:10 PM] fearn Downtime
fearn (www40) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Jun 13, 2001, 5:01 PM] vepar Downtime
vepar (www201) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Jun 12, 2001, 1:30 AM] whury Downtime
whury crashed under heavy load and had to be manually rebooted. Total
downtime was less then 15 minutes.
[Jun 11, 2001, 3:46 PM] OpenSRS Outage
In addition to the slowed response at the central registry we reported
earlier, the OpenSRS system is suffering an outage at this time.
OpenSRS support has acknowledged the outage, but could offer no explanation
or an ETA of when service will be restored. While the outage continues,
domain registrations and management via the Account Control Center, the
signup system, and pairnic.com will be unavailable. We will post more
information as it becomes available.
[Jun 11, 2001, 8:02 AM] Registry Under Heavy Load
Verisign GRS has advised us that the central domain registry is under
unusually heavy load this morning, and that all registrars can expect
slow service until the load is addressed. If further information becomes
available, we will post again.
[Jun 10, 2001, 6:58 PM] eahta Downtime
Due to a user script problem, eahta (www227) crashed twice between 6pm
and 7pm Eastern time. The problem was identified and corrected, and
service has returned to normal.
[Jun 8, 2001, 1:07 AM] quesse Maintenance Completed
The hard drive swap for quesse (44) has been completed. Total downtime for
the maintenance was approximately 10 minutes.
[Jun 7, 2001, 10:52 PM] quesse Emergency Maintenance
We have detected errors in the hard drive of server quesse (44). We will
be performing a hard drive swap on the server shortly. There will be two
periods of downtime which should each be five minutes or less. We will post
more information as it becomes available.
[Jun 7, 2001, 2:12 PM] niwun Problems
Over the past hour, web sites on niwun (www225) were intermittently
unresponsive due to an infinite loop inadvertently caused by a user who was
using Apache's mod_rewrite module. We are working directly with the user to
ensure that the problem does not recur.
[Jun 7, 2001, 11:59 AM] dhaame Downtime
dhaame (www77) crashed under high load and had to be rebooted. Total
downtime was about 10 minutes.
[Jun 5, 2001, 1:32 PM] diviy Downtime
diviy (www217) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Jun 4, 2001, 2:09 PM] db19 Emergency Maintenance Completed
We have completed the hard drive swap of db19.pair.com. We expect that the
server will have no further issues. Total downtime for the swap was
approximately 10 minutes.
[Jun 4, 2001, 12:44 PM] db19 Emergency Maintenance
We have detected errors on the hard drive in db19.pair.com. We will be
performing a hard drive swap on the server shortly. There will be two
periods of downtime, which should each be five minutes or less. We will
post additional information as it becomes available.
[Jun 4, 2001, 12:35 PM] vepar Downtime
vepar (www201) hung under high load, and was restarted. It has since return
to normal service.
[Jun 3, 2001, 3:32 AM] Network Problems Resolved
The problems with the router have been resolved as of 3:27 AM. During the
time of this outage traffic attempting to get to sites hosted on our
servers would have had some difficulties, depending on which backbone they
were coming in on.
[Jun 3, 2001, 3:16 AM] Network Problems
We are currently experiencing problems with one of our routers, this has
been the case since 2:50 AM. We are currently in the process of bringing it
back online, and an additional notice will be posted when the router is
back online.
[Jun 2, 2001, 9:33 PM] BIOS updates
The following servers received BIOS updates tonight:
bellat (www101)
eahta (www227)
petora (www229)
xebte (www230)
db16.pair.com
db18.pair.com
db20.pair.com
Total downtime per server was approximately two minutes.
[Jun 1, 2001, 2:04 PM] laguz Downtime
laguz (www148) crashed under heavy load, and was reboot. It has since
returned to normal operations.
[Jun 1, 2001, 3:36 AM] emlis Maintenance Completed
Maintenance on emlis.pair.com (www216) has been completed. We apologize for the
downtime, and any inconvenience.
[Jun 1, 2001, 1:19 AM] emlis Emergency Maintenance
User server emlis (www216) has suffered a hard drive failure and will be
down for replacement and maintenance. Total downtime is not expected to
exceed 90 minutes, and any lost data will be restored from backups.
[May 31, 2001, 1:19 PM] Network Problems Resolved
During routine reconfiguration of our network to accommodate new IP
addresses issued by ARIN, traffic from some providers to domains hosted
on 66.39.x.x IP addresses was briefly blocked. The duration of the
problem was approximately ten minutes, and also affected our main Web
site www.pair.com. This has been resolved and should not recur.
Please accept our apologies for the inconvenience.
[May 30, 2001, 1:40 PM] foma Downtime
foma (www176) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[May 29, 2001, 4:52 PM] Network Problems Resolved
Sprint has corrected their network maintenance, and we have not see any
more dops in traffic for some time. All traffic is flowing normally again
now as of 2:30.
[May 29, 2001, 4:01 PM] diviy Downtime
diviy (www217) was rebooted to clear up a resource shortage problem, which
was affecting the server's useability. Downtime was approximately one
minute.
[May 29, 2001, 2:47 PM] Network Problems
Beginning at approximately 11 am this morning, we have observed periodic
drops in traffic on our Sprint circuit. Sprint has reported having problems
in their network today, which are causing these intermittent drops. Each
drop is very brief, so impact on traffic should be very small.
We are monitoring the situation with Sprint until these troubles are
resolved.
[May 29, 2001, 2:07 PM] OpenSRS Back Online
OpenSRS appears to have returned their systems to normal operation at this
time. We have therefore restored all domain services in the signup system,
Account Control Center, and on pairnic.com. We will continue to seek
further details as to the cause of this unfortunate incident. The length
of their outage was approximately 15 hours.
We would like to again note that this outage did not affect the operation
of any existing domain registration. All registered domains continued to
have normal nameservice, mail delivery, and Web hosting.
[May 29, 2001, 10:53 AM] OpenSRS Outage
We have been advised by OpenSRS, our current domain registrar service,
that they have been experiencing what they refer to simply as "technical
problems", beginning around 10:30pm Eastern time yesterday. They have
further indicated that they expect to resolve the problem "within a few
hours." Until their service is restored, we have suspended our account
signup system and domain registration services.
We apologize for the tremendous inconvenience to our customers and
resellers. We are extremely disappointed with their handling of this
situation, including not only the apparent technical limitations which
may have played a role in the downtime, but the belated announcement of
the problem with no useful information regarding its cause or resolution.
We are accelerating our work towards direct registrar operations, in order
to reduce dependence on third-party performance.
Please note that this outage does not affect the operation of any existing
domain registration, except that WHOIS will be unavailable, and domain
registrations and modifications cannot be performed. Registered domains
will continue to have normal nameservice, mail delivery, Web hosting, and
other related services.
[May 29, 2001, 8:21 AM] OpenSRS Problems
We are currently unable to reach the servers at OpenSRS, our registrar
partner, affecting all pairNIC domain registrations. This problem appears
to be affecting all OpenSRS partners. We will post more information as it
becomes available.
[May 25, 2001, 1:09 PM] db11 Emergency Maintenance Completed
We have completed the chassis swap of db11.pair.com. The server is now a
Celeron 800 MHz, with 256 MB of RAM. We full expect this swap will resolve
any previously lingering issues with the server. Downtime for the swap was
approximately five minutes.
[May 25, 2001, 12:40 PM] db11 Emergency Maintenance
We have unfortunately tracked some hardware problems with the new chassis
for db11.pair.com. We will be swapping db11.pair.com into a new chassis on
an emergency basis. We will post additional information as it becomes
available.
[May 25, 2001, 7:30 AM] Verisign Registry Downtime
Verisign has advised all domain registrars that the central domain
registry will be down for emergency maintenance between 2pm and 3:30pm
Eastern time today, Friday May 25.
During this time, new domain registrations and transfers will be
unavailable through our signup system, the Account Control Center,
and the pairNIC Web site.
[May 25, 2001, 6:32 AM] db11 Maintenance Completed
We have completed the hard drive swap on db11, and will continue to monitor
the server closely for any further problems.
[May 25, 2001, 5:31 AM] db11 Problems
We have determined that due to an apparent failure in the db11 drive
controller since its recent upgrade, the MySQL service has not been
running reliably. We will be taking the server down for an emergency swap
effectively immediately. We expect it to be restored to full service
within the next hour. Please accept our apologies for the inconvenience.
[May 25, 2001, 1:41 AM] BIOS updates
The following user servers received BIOS updates tonight:
kappa (www6)
omicron (www9)
upsilon (www13)
chi (www14)
psi (www15)
dair (www20)
tinne (www21)
fearn (www40)
anga (www47)
lambe (www63)
wredhor (www76)
nish (www92)
vuae (www93)
sether (www95)
vert (www102)
tenshian (www103)
dhabat (www105)
elhaz (www144)
diaf (www215)
fimfi (www223)
Total downtime per server ranged from 10 - 15 minutes. They have all been
restored to normal service.
[May 24, 2001, 2:09 PM] db11 Maintenance Completed
The maintenance of db11.pair.com has been completed. In addition to having
upgraded versions of FreeBSD and MySQL, the hardware was also upgraded to
an AMD Duron 800 MHz, with 256 MB of RAM. Total downtime was approximately
15 minutes for two outages.
[May 24, 2001, 12:02 PM] db13 Maintenance Completed
The maintenance on db13.pair.com has been completed. Total downtime was
approximately 10 minutes.
[May 24, 2001, 11:26 AM] vuae Downtime
vuae (www93) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[May 23, 2001, 10:44 PM] oynos Maintenance Completed
We have completed the drive swap of oynos (www218). Total downtime for this
maintenance was about 10 minutes.
[May 23, 2001, 10:26 PM] BIOS updates
The following user servers received BIOS updates tonight:
iota (www5)
delsi (www206)
kodh (www90)
oynos (www218)
halla (www68)
yanta (www67)
hyarmen (www66)
flaid (www91)
omega (www16)
hwesta (www51)
umbar (www46)
ando (www45)
onn (www29)
beeoro (www69)
zhun (www107)
Total downtime per server ranged from 10 - 15 minutes. They have all been
restored to normal service.
[May 23, 2001, 7:47 PM] oynos Emergency Maintenance
We have observed a warning condition on the hard drive in oynos (www218)
which may indicate an imminent failure. We will be taking the server
offline shortly to initiate a drive swap. There will be two brief periods
of downtime, less than ten minutes each. An additional notice will be
posted once the swap is complete.
[May 23, 2001, 6:19 PM] db2, db17 Maintenance Completed
The maintenance on db2.pair.com and db17.pair.com has been completed.
Downtime was approximately 10 minutes per server.
[May 23, 2001, 5:52 PM] db11, db13 Planned Maintenance
We are planning to begin maintenance on db11.pair.com and db13.pair.com, at
approximately 11:00 AM Eastern, Thursday, May 24. This includes upgrading
FreeBSD and MySQL. There will be two periods of downtime, which should be
no longer than ten minutes each. We will post additional notices as
appropriate.
[May 23, 2001, 1:26 PM] db9, db10 Maintenance Completed
The upgrades of db9.pair.com and db10.pair.com have been completed. The
servers are now running the most recent versions of MySQL and FreeBSD.
Total downtime was approximately 10 minutes for each server.
[May 23, 2001, 12:08 PM] db2, db17 Emergency Maintenance
We will be taking down db2.pair.com and db17.pair.com briefly after 6PM
to perform a BIOS upgrade. The total downtime for this maintenance will be
between 10 and 15 minutes per server. We will post an additional notice
when this upgrade has been completed.
[May 22, 2001, 6:09 PM] db Maintenance Completed
The maintenance on db.pair.com has been completed. Downtime was
approximately five minutes.
[May 22, 2001, 3:02 PM] tehun Downtime
tehun (www226) crashed under heavy load and was rebooted. The downtime was
under 15 minutes.
[May 22, 2001, 2:55 PM] db Emergency Maintenance
We will be taking down db.pair.com briefly after 6PM to perform a BIOS
upgrade. The total downtime for this maintenance will be between 10 and 15
minutes. We will post an additional notice when this upgrade has been
completed.
[May 22, 2001, 2:37 PM] uilen Downtime
uilen (www35) crashed under heavy load and needed to be rebooted. The
downtime was under 20 minutes.
[May 22, 2001, 10:35 AM] foma Downtime
foma (www176) crashed under heavy load and had to be rebooted. Total
downtime was around 10 minutes.
[May 22, 2001, 10:07 AM] db9 and db10 Planned Maintenance
We are planning to begin maintenance on db9.pair.com and db10.pair.com, at
approximately 11:00 AM Eastern, Wednesday, May 23. This includes upgrading
FreeBSD and MySQL. There will be two periods of downtime, which should be
no longer than ten minutes each. We will post additional notices as
appropriate.
[May 22, 2001, 2:34 AM] jera Downtime
jera (www142) crashed under heavy load and had to be rebooted. Total
downtime was about 10 minutes.
[May 21, 2001, 7:06 PM] dhabat Maintenance Completed
dhabat (www105) is back in service after undergoing drive maintenance.
Total downtime was 5 minutes.
[May 21, 2001, 7:03 PM] anca Maintenance Completed
We have completed the BIOS upgrade of anca (www53). We believe this upgrade
will take care of some lingering issues on this server. Total downtime for
this maintenance was about 10 minutes.
[May 21, 2001, 6:49 PM] dhabat Emergency Maintenance
dhabat (www105) will be going down shortly to correct a possible problem
which was detected on the hard drive. Downtime should be no longer than 15
minutes.
[May 21, 2001, 5:53 PM] Network Problems Resolved
As of 5pm, the problems on SAVVIS' network have been resolved. We have not
received an official explanation, but traffic is now flowing normally.
[May 21, 2001, 4:27 PM] anca Emergency Maintenance
We will be taking anca (www53) down briefly after 6PM to perform a BIOS
upgrade. The total downtime for this maintenance will be between 10 and 15
minutes. We will post an additional notice when this upgrade has been
completed.
[May 21, 2001, 4:08 PM] Network Problems
Beginning at approximately 3pm Eastern time, SAVVIS has been having
difficulty passing traffic through network, resulting in reduced
throughput, fewer route announcements, and intermittent periods of no
connectivity through SAVVIS, 10-20 seconds at a time.
We are attempting to determine the official cause of the problem. If it
persists, we will shift traffic away from SAVVIS to other providers. More
information will be provided as it becomes available.
[May 21, 2001, 2:23 AM] chi Downtime
chi(www14) crashed under heavy load and had to be rebooted. Downtime was
about 10 minutes.
[May 20, 2001, 2:34 AM] voot Downtime
voot crashed under heavy load and had to be rebooted. Total downtime was
less then 10 minutes.
[May 19, 2001, 9:32 PM] Registry Downtime
Verisign is currently in a maintenance window. Until approximately 1am
Eastern time Sunday morning, we do not expect that domain registrations
will work correctly. We apologize for the late notice. Please defer any
new registrations or account signups until after 1am.
[May 17, 2001, 2:21 PM] db7.pair.com maintenance completed
The upgrade of db7.pair.com has been completed. The server is now running
the most recent versions of MySQL and FreeBSD. Total downtime was
approximately 10 minutes.
[May 17, 2001, 2:00 PM] db8 maintenance completed
The upgrade of db8.pair.com has been completed. The server is now running
the most recent versions of MySQL and FreeBSD. Total downtime was
approximately 10 minutes.
[May 17, 2001, 12:29 AM] thuule Maintenance Completed
We have completed the emergency drive swap of thuule (www49). It now has a
20GB drive. Total downtime for this maintenance was about 10 minutes.
[May 16, 2001, 9:51 PM] thuule Emergency Maintenance
We have observed a warning condition on the hard drive in thuule (www49)
which may indicate an imminent failure. Consequently, we will be taking
this server down shortly to initiate a drive swap. There will be two brief
periods of downtime, less than ten minutes each. We will post an additional
notice when the swap has been completed.
[May 16, 2001, 2:52 PM] db7, db8 Planned Maintenance
We are planning to begin maintenance on db7.pair.com and db8.pair.com, at
approximately 11:00 AM Eastern, Thursday, May 17. This includes upgrading
FreeBSD and MySQL. There will be two periods of downtime, which should be
no longer than ten minutes each. We will post additional notices as
appropriate.
[May 16, 2001, 2:14 PM] anca Downtime
anca (www53) crashed under heavy load, and had to be rebooted. Total
downtime was approximately 15 minutes.
[May 16, 2001, 2:12 PM] db5 Maintenance Completed
The upgrade of db5.pair.com has been completed. The server is now running
updated versions of MySQL and FreeBSD. Total downtime was approximately
ten minutes.
[May 16, 2001, 1:22 PM] db6 Maintenance Completed
The upgrade of db6.pair.com has been completed. The server is now running
updated versions of MySQL and FreeBSD. Additionally, more RAM was added
to bring the total to 256 MB. Total downtime was approximately ten minutes.
[May 16, 2001, 10:12 AM] gebo Downtime
gebo (www139) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[May 16, 2001, 8:53 AM] Recent Spam Incidents
We would like to take a moment to reassure our customers that at no point
do we provide their account information or e-mail address to any third
party, except that WHOIS information is available through domain
registrations.
Recently, an external Spammer artificially created a huge mailing list of
@pair.com addresses, simply by prepending a huge list of plausible account
names. The resulting flood of e-mails was noticed by our systems and
partially purged before it could complete delivery, but nonetheless, many
valid accounts received e-mail which appeared to be addressed to them
specifically. We would like to emphasize that this was a scattershot
mailing done by an unauthorized third party, and at no point have our
customer lists been revealed.
We are also pursuing action on other Spam incidents, both through FormMail
and other methods, which appear designed to impugn our anti-Spam reputation.
Needless to say, our position on unsolicited e-mail has not changed.
[May 15, 2001, 1:30 PM] db5, db6 planned maintenance
We are planning to begin maintenance on db5.pair.com and db6.pair.com, at
approximately 11:00 AM Eastern, Wednesday, May 16. This includes upgrading
FreeBSD and MySQL. There will be two periods of downtime, which should be
no longer than ten minutes each. We will post additional notices as
appropriate.
[May 15, 2001, 1:03 PM] db.pair.com Maintenance Completed
The upgrade of db.pair.com has been completed. The server is now running
updated versions of MySQL and FreeBSD. Additionally, the hardware has been
upgraded to an AMD Athlon 1000 MHz, with 256 MB of RAM. Total downtime was
approximately ten minutes.
[May 15, 2001, 7:37 AM] voot Downtime
voot (www154) crashed under heavy load and had to be manuall rebooted.
Total downtime was approximately 15 minutes.
[May 14, 2001, 1:34 PM] diviy Downtime
diviy (www217) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[May 14, 2001, 11:54 AM] db.pair.com planned maintenance
We are planning to begin maintenance on db.pair.com, at approximately
11:00 AM Eastern, Tuesday, May 15. This includes upgrading
FreeBSD, MySQL, and the hardware itself. There will be two periods of
downtime, which should be no longer than ten minutes each. We will post
additional notices as appropriate.
[May 14, 2001, 5:05 AM] Network Problems Resolved
Beginning at 2:15am and continuing for approximately thirty minutes,
Sprint's network apparently suffered route propagation problems and
blackholed some of our traffic for some Sprint peers. The situation
was corrected promptly and traffic has returned to normal levels.
[May 11, 2001, 1:28 PM] db3 and db4 maintenance completed
The maintenance on db3.pair.com and db4.pair.com has been completed. The
servers received an upgrade of FreeBSD and MySQL. Total downtime for each
server was approximately 15 minutes.
[May 11, 2001, 5:36 AM] emancholl Downtime
emancholl crashed under heavy load and had to be rebooted. Downtime was
less than 10 minutes.
[May 10, 2001, 4:39 PM] diviy Load
We have been tracking high load conditions on diviy (www217). In an
unfortunate coincidence, this server has a remarkable number of busy sites.
Several accounts are being moved to other servers, as well as being offered
QuickServe dedicated hosting. We expect an improvement in server load as
early as Friday.
[May 10, 2001, 3:47 PM] db3 and db4 planned maintenance
We are planning to begin maintenance on db3.pair.com and db4.pair.com, at
approximately 11:00 AM Eastern, Friday, May 11. This includes upgrading
FreeBSD, MySQL, and the hardware itself. There will be two periods of
downtime, which should be no longer than ten minutes each. We will post
additional notices as appropriate.
[May 10, 2001, 3:25 PM] db2 maintenance completed
The maintenance on db2.pair.com has been completed. The server is running
the most recent versions of MySQL and FreeBSD. Also, the hardware is now
an Athlon 1000 MHz, with 256 MB RAM. Downtime was approximately 10
minutes.
[May 10, 2001, 2:38 PM] fearn Downtime
fearn (www40) crashed under heavy load and had to be rebooted total
downtime was approximately 10 minutes.
[May 10, 2001, 2:05 AM] Network Problems Resolved
Around 1:45am Eastern time, a low memory condition caused routing problems
on one of our internal gateway routers. For approximately fifteen minutes,
traffic from some servers was unable to pass out of our network. We have
corrected the condition. These routers are being phased out of our network
within the next two to three weeks. We apologize for the disruption.
[May 9, 2001, 8:48 PM] akket Maintenance Completed
We have completed the drive maintenance of akket (www211). Total downtime
for this maintenance was about 10 minutes.
[May 9, 2001, 5:54 PM] akket Emergency Maintenance
We have observed a warning condition on the hard drive in akket (www211)
which may indicate an imminent failure. We will be taking the server
offline after 7pm Eastern time to initiate a drive swap. There will be
two brief periods of downtime, less than ten minutes each. We will post an
additional notice after the swap has been completed.
[May 9, 2001, 1:32 PM] db2 maintenance
The maintenance on db2.pair.com has been started. Server downtime was
under five minutes. There will be another period of downtime when the
maintenance is completed. We will post additional notices with more
information as appropriate.
[May 8, 2001, 4:48 PM] athani Downtime
athani (www157) crashed under heavy load and had to be rebooted. Total
downtime was approximately ten minutes.
[May 8, 2001, 3:17 PM] db2 planned maintenance
We are planning to start an upgrade of db2.pair.com, at approximately
1:30 PM, Wednesday, May 9. There will be two periods of downtime, under five
minutes each. The operating system, MySQL software, and server hardware will
all be upgraded. We will post additional notices with further information,
when appropriate.
[May 8, 2001, 1:35 PM] diviy Downtime
diviy (www217) developed a high load condition which required the server to
be rebooted. Downtime was approximately 10 minutes.
[May 8, 2001, 2:02 AM] db Server Problem
Beginning at 1am Eastern time, our db.pair.com server went offline. The
problem was traced to a faulty Ethernet card, which has been replaced. We
will be upgrading these servers in the very near future. Total downtime
was approximately 50 minutes.
[May 7, 2001, 9:15 PM] naam Maintenance Completed
We have completed the drive maintenance of naam (www114). We have also
swapped it's hardware. It is now a Pentium III 1000MHz with 256MB of RAM
and a 30GB disk. Total downtime for this maintenance was about 10 minutes.
[May 7, 2001, 2:58 PM] diviy Downtime
diviy (www217) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[May 7, 2001, 1:41 PM] naam Planned Maintenance
We have observed a warning condition on the hard drive in naam (www114)
which may indicate an imminent failure. Consequently, this evening after
6pm Eastern time, we will be taking the server offline to initiate a drive
swap. There will be two brief periods of downtime, less than ten minutes
each. We will include a upgrade to CPU and memory capacity as well.
We will post an additional notice when this maintenance has been completed.
[May 7, 2001, 7:49 AM] yekk Maintenance Completed
Both the primary hard drive and the primary IDE controller on yekk.pair.com
failed. We prepared a new chassis for yekk.pair.com, and have rebuilt the
hard drive from backups. The server has been returned to normal operation,
and is now running as a Pentium III 1Ghz with 256M RAM.
[May 7, 2001, 2:33 AM] yekk Emergency Maintenance
The hard drive on yekk (www124) has failed as of 1:45am Eastern time. We
are working to restore the server to full operation; this may require a
restore from backups. We expect to have the matter resolved within the
hour. We will post further information as it becomes available.
[May 6, 2001, 8:49 AM] psi Logs May 5
Due to a data corruption error, Web logs for May 5 are not available on
psi (www15). Please accept our apologies for the inconvenience.
[May 4, 2001, 9:12 AM] OpenSRS Timeouts
As posted yesterday, we continue to see an intermittent problem with
OpenSRS, our registrar partner. Their servers are in many cases
timing out without responding to requests made to them. We have been
advised that they are working on the problem, but have not yet been
given specific details or a timeframe for resolution.
The continued impact of this problem is that new domain registrations, whois
lookups, and certain domain management functions may fail or timeout until
this is resolved. We will continue to press them for an expedited
resolution, and will post more details if they become available.
[May 3, 2001, 3:04 PM] diviy Downtime
diviy (www217) crashed under heavy load, and was brought back online.
Downtime was approximately 10 minutes.
[May 3, 2001, 9:33 AM] OpenSRS Timeouts
We have been tracking an intermittent problem with OpenSRS, our registrar
partner, this morning. For brief periods, their service is not responding.
We have not yet determined if this is a problem with their network,
servers, or software. We will post more information as it becomes
available.
The impact of this is that new domain registrations, whois lookups, and
certain domain management functions may occasionally fail or timeout until
this problem is resolved.
[May 2, 2001, 9:36 PM] Network Problems Resolved
We have brought our new DS-3 circuit online to Digex/Intermedia. The
new circuit eliminates SMDS overhead and has greater overall capacity as
a result. We do not expect any further downtime, but will monitor this
new circuit very closely. Some users may experience an improvement in
performance.
[May 2, 2001, 8:35 PM] Network Problems
We are currently working to resolve an administrative misunderstanding
with Digex/Intermedia. As of 7pm Eastern time, they have disabled our
DS-3 circuit, on the basis of some automated procedure, in spite of the
fact that our newer DS-3 circuit with them, which we have been trying to
bring online, is not yet in service.
Currently neither circuit is operating, and the traffic is being carried
by alternate providers. We will restore at least one Digex/Intermedia
circuit to proper operation as soon as possible this evening, and post
further information as it becomes available.
[May 2, 2001, 5:24 PM] niwun reboot
niwun (www225) was rebooted to clear up a high load condition. Downtime was approximately one minute.
[May 2, 2001, 10:37 AM] omicron Downtime
omicron (www9) crashed under heavy load and had to be rebooted. The total
downtime was approximately 10 minutes.
[Apr 30, 2001, 10:12 AM] iota Downtime
iota (www5) crashed under heavy load and had to be
rebooted. Total downtime was approximately 10 minutes.
[Apr 29, 2001, 7:57 AM] iota Downtime
A resource starvation problem was caused on iota this morning by a runaway
user script. The problem has been tracked down and eliminated. Some users
may have received slow service on iota between 5 and 7:30am Eastern time.
[Apr 27, 2001, 5:20 PM] vepar Downtime
vepar (www201) crashed under high load and had to be rebooted. Total downtime was ten minutes.
[Apr 25, 2001, 6:26 PM] arda Maintenance Completed
We have completed the drive upgrade of arda (www62). It now has a 30GB
disk. Total downtime for this upgrade was about 10 minutes.
[Apr 25, 2001, 6:05 PM] nwalme Maintenance Completed
We have completed the drive upgrade of nwalme (www57). It now has a 30GB
disk. Total downtime for this upgrade was about 10 minutes.
[Apr 25, 2001, 6:49 AM] Network Problems Resolved
Our Sprint DS-3 has been restored to normal operation. At all times during
the outage, traffic was carried via alternate providers.
[Apr 25, 2001, 6:10 AM] Network Problems
Our Sprint DS-3 has gone out of service approximately twenty minutes ago. We
are working with them to bring the circuit back online. In the meantime, traffic
continues to flow normally through our other providers. We will post more
information as it becomes available.
[Apr 25, 2001, 2:05 AM] tinne Maintenance Completed
The drive swap on tinne (www21) has been completed. All services on this
server should again be functioning as normal.
[Apr 24, 2001, 11:40 PM] tinne Emergency Maintenance
A hard drive swap will be done on tinne (www21) momentarily. There will be
two brief periods of downtime of about 5 minutes while this is done. We
will post an additional notice once this maintenance is completed.
[Apr 24, 2001, 8:23 PM] dhabat Downtime
dhabat (www105) crashed under heavy load. It required a manual file system
check before being brought back online. Downtime was 20 minutes.
[Apr 24, 2001, 7:51 PM] gao Maintenance Completed
We have completed the RAM upgrade of gao (www111). It now has 384MB or RAM.
Total downtime for this upgrade was about 5 minutes.
[Apr 24, 2001, 7:22 PM] yekk, db2, db4, db5 Maintenance Completed
We have completed the ethernet upgrades of yekk (www124), db2, db4 and db5.
Total downtime for each server was about 5 minutes.
We have postponed the ethernet upgrade of db.pair.com to a later date.
[Apr 24, 2001, 6:48 PM] unque Maintenance Completed
We have completed the drive upgrade of unque (www54). It now has a 30GB
disk. Total downtime for this maintenance was about 10 minutes.
[Apr 24, 2001, 6:23 PM] arda Planned Maintenance
We are about to begin a drive upgrade on arda (www62) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 24, 2001, 6:13 PM] nwalme Planned Maintenance
We are about to begin a drive upgrade on nwalme (www57) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 24, 2001, 5:58 PM] noldo Maintenance Completed
We have completed the drive upgrade of noldo (www56). It now has a 30GB
disk. Total downtime for this maintenance was about 10 minutes.
[Apr 24, 2001, 3:09 PM] derba, gao Planned Maintenance
We will be adding more RAM to gao (www111) tonight. This upgrade will
require about 5 minutes of downtime. This maintenance will occur between
7:30PM and 8:30PM EST tonight. An additional notice will be posted when
the upgrade is completed.
We will also be taking derba (www104) down breifly to tighten it's video
card. This maintenance should take about five minutes, and will occur
between 7:30PM and 8:30PM EST tonight.
[Apr 24, 2001, 3:05 PM] yekk, db, db2, db4, db5 Planned Maintenance
We will be taking down yekk (www124), db.pair.com, db2.pair.com,
db4.pair.com and db5.pair.com breifly tonight to upgrade their ethernet
cards. This maintenance will happen between 6:30PM and 8:00PM EST. Total
downtime for each server should be about 10 minutes. An additional notice
will be posted when the maintenance on each server is complete. This
upgrade will not affect user accounts.
[Apr 23, 2001, 4:19 PM] fearn Downtime
fearn (www40) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Apr 23, 2001, 3:59 PM] emancholl Downtime
emancholl (www37) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 10 minutes.
[Apr 23, 2001, 11:56 AM] fearn Downtime
fearn (www40) crashed again shortly after the first crash, and was quickly
rebooted. The source of the problem was traced to a mailing error,
which caused the server to crash the second time. This was manually
cleared up before the server was fully rebooted the second time, and the
server is now running smoothly. Total downtime was approximately 20
minutes.
[Apr 23, 2001, 11:36 AM] fearn Downtime
fearn (www40) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 20 minutes.
[Apr 22, 2001, 10:14 PM] noldo Planned Maintenance
We are about to begin a drive upgrade on noldo (www56) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 22, 2001, 9:59 PM] unque Planned Maintenance
We are about to begin a drive upgrade on unque (www54) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 22, 2001, 9:56 PM] ampa Maintenance Completed
We have completed the drive upgrade of ampa (www52). It now has a 30GB
disk. Total downtime for this maintenance was about 10 minutes.
[Apr 22, 2001, 9:30 PM] lambe Planned Maintenance
We have completed the drive upgrade of lambe (www63). It now has a 30GB
disk. Total downtime for this maintenance was about 10 minutes.
[Apr 22, 2001, 8:50 PM] hwesta Maintenance Completed
We have completed the drive upgrade of hwesta (www51). It now has a 30GB
disk. Total downtime for this maintenance was about 10 minutes.
[Apr 22, 2001, 6:28 PM] lambe Planned Maintenance
We are about to begin a drive upgrade on lambe (www63) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 22, 2001, 6:01 PM] ampa Planned Maintenance
We are about to begin a drive upgrade on ampa (www52) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 22, 2001, 5:49 PM] hwesta Planned Maintenance
We are about to begin a drive upgrade on hwesta (www51) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance. The
server will remain online at all other times. The entire upgrade could take
up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Apr 22, 2001, 1:19 AM] halla Downtime
halla crashed under high load and had to be rebooted. Downtime was less
than 10 minutes.
[Apr 21, 2001, 8:39 AM] Server Log Problems
On approximately two dozen servers, Web logs were not generated overnight
due to a configuration error. This has been corrected, and those logs are
being generated at this time. Some customers may therefore have experienced
a delay in receiving their Web logs overnight.
[Apr 18, 2001, 7:37 PM] omega Maintenance Completed
We have completed the hardware swap of omega (www16). It is now an Athlon
Thunderbird 1000MHz, with 256MB of RAM. The total downtime for this
maintenance was about 10 minutes.
[Apr 18, 2001, 7:19 PM] upsilon Maintenance Completed
We have completed the hardware swap of upsilon (www13). It is now an Athlon
Thunderbird 1000MHz, with 256MB of RAM. The total downtime for this
maintenance was about 20 minutes.
[Apr 18, 2001, 2:27 PM] upsilon, omega Planned Maintenance
Due to suspected hardware problems, we will be swapping upsilon (www13) and
omega (www16) to new hardware this evening, between 6pm and 8pm EST. Total
downtime for each server should be about 10 minutes. We will post an
additional notice after the swap.
[Apr 17, 2001, 3:25 PM] Network Problems Resolved
We have completed the migration of our Sprint DS-3 circuit to alternate
equipment. This has also reduced congestion in our network, while allowing
for the pending OC-3c upgrade to Sprint. There are no observed performance
problems or outages at this time. We appreciate our customers' patience
during this brownout. We will continue to monitor for any further
problems.
[Apr 17, 2001, 2:58 PM] csor Downtime
csor (www190) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Apr 17, 2001, 2:43 PM] bemnet Downtime
bemnet (www158) crashed under heavy load and had to be rebooted. Total
downtime was approximately 15 minutes.
[Apr 17, 2001, 2:01 PM] Network Problems
We have replaced cabling and equipment on our Sprint DS-3 circuit, and
are working to reconfigure the new equipment for optimal performance.
There is minor saturation on the Sprint circuit at this time, but all
circuits are working normally otherwise.
[Apr 17, 2001, 11:02 AM] Network Problems
Our Sprint DS-3 equipment has gone out of service again, and we are
working with Sprint engineers to determine the resolution. Our traffic
is being carried by alternate providers. We will post further information
as it becomes available.
[Apr 17, 2001, 10:30 AM] csor Downtime
The server crashed under high load and had to be rebooted.
Downtime was about 10 minutes.
[Apr 17, 2001, 9:47 AM] Network Problems Resolved
We have restored our Sprint DS-3 to normal operation. We suspect a faulty
cable, which will be replaced within the next 48 hours. There will be
another brief outage at that time, during which alternate providers will
carry our traffic with no loss.
[Apr 17, 2001, 8:52 AM] Network Problems
Our Sprint DS-3 has gone out of service approximately ten minutes ago.
We are working with their engineers to determine the cause. In the
meantime, traffic continues to flow normally through alternate providers.
We will post more information as it becomes available.
[Apr 16, 2001, 11:46 PM] kodh Downtime
kodh crashed under high load adn had to be rebooted. Downtime was about
10 minutes.
[Apr 16, 2001, 10:11 AM] psi Downtime
psi (www15) crashed under heavy load, and returned after a filesystem
check.
[Apr 16, 2001, 4:23 AM] aedde Downtime
aedde crashed under high load and had to be rebooted. Downtime was about
10 minutes.
[Apr 14, 2001, 12:24 AM] chi Downtime
chi crashed under high load and had to be rebooted. Downtime was about
10 minutes.
[Apr 12, 2001, 1:26 PM] bemnet reboot
Due to a resource shortage, bemnet (www158) was rebooted. Downtime was
approximately one minute.
[Apr 12, 2001, 3:59 AM] iota Downtime
iota chrashed under high load and had to be rebooted. Downtime was less
then 10 minutes.
[Apr 12, 2001, 2:48 AM] bellat Maintenance Completed
The hard drive swap on bellat (www101) has been completed. To ensure that
the problem was corrected we also did a simultaneous motherboard swap. The
server is now again up and running.
[Apr 11, 2001, 11:25 PM] bellat Emergency Maintenance
bellat (101) will be going down for an emergency hard drive swap
momentarily. We will post again once the swap has been completed.
[Apr 11, 2001, 11:48 AM] Network Problems Resolved
We have worked with engineers from Sprint, UUnet, and Digex to identify
and block a denial-of-service attack against our network this morning.
We are working further to eliminate any possible future impact of this
type of attack. Our thanks are extended to all engineers who have
assisted us, as well as customers who have tolerated the impact on
traffic. We were able to reroute traffic across our UUnet shadow DS-3
during most of the attack, thereby reducing the impact.
[Apr 11, 2001, 10:21 AM] Network Problems
We are experiencing stability problems with our OC-3c circuit to UUnet.
We are working with UUnet engineers to resolve this, and will post more
information as it becomes available.
[Apr 11, 2001, 8:55 AM] kodh Downtime
kodh (www90) crashed under load and was brought back online after a
filesystems check. Downtime was approximately 15 minutes.
[Apr 10, 2001, 8:59 PM] ceirt Downtime
ceirt (www39) crashed under high load and had to be rebooted. Total
downtime was about 10 minutes.
[Apr 10, 2001, 7:34 PM] cuzea Downtime
cuzea (www94) crashed under heavy load and had to be rebooted. Total
downtime was about 10 minutes.
[Apr 10, 2001, 12:36 PM] Intermittent Domain Management Troubles
Due to a problem with the OpenSRS system, which they report
has been corrected, some users may have experienced
intermittent trouble this morning using an existing OpenSRS
profile for domain management. Anyone who is still experiencing
trouble should let us know at support@pairnic.com.
[Apr 9, 2001, 1:01 PM] laguz Downtime
laguz (www148) crashed under heavy load and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Apr 9, 2001, 11:47 AM] anca Downtime
anca (www53) crashed under heavy load, and the filesystem had to be
manually cleaned before it could be rebooted. Total downtime was
approximately 15 minutes.
[Apr 8, 2001, 11:16 PM] ciot Downtime
ciot (www161) crashed under high load and had to be rebooted. Total
downtime of about 15 minutes.
[Apr 5, 2001, 9:16 PM] oomur Maintenance Completed
We have completed the drive swap of oomur (www209). It now has a 30GB disk,
and we expect no further complications from this drive. Total downtime for
this swap was about 10 minutes.
[Apr 5, 2001, 5:56 PM] ilceille Downtime
ilceille (www189) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Apr 5, 2001, 5:52 PM] oomur Emergency Maintenance
We are about to begin a drive swap on oomur, due to errors on the current
drive. There will be two brief periods of downtime, each approximately five
minutes, at the beginning and end of the maintenance. The server should
remain online at all other times.
We will post an additional notice when the upgrade has been completed.
[Apr 5, 2001, 2:55 PM] kodh Downtime
kodh (www90) crashed under heavy load and had to be rebooted. Total
downtime was approximately 15 minutes.
[Apr 4, 2001, 12:22 AM] kenaz Downtime
kenaz (www138) crashed under high load and had to be rebooted.
[Apr 3, 2001, 3:10 PM] zulle Downtime
zulle (www175) crashed under heavy load and had to be manually reboot.
Total downtime was approximately 10 minutes.
[Apr 3, 2001, 2:52 PM] kodh Downtime
kodn (www90) crashed under heavy load, and had to have a thorough disk
cleaning before being rebooted. Total downtime was approximately 20
minutes.
[Apr 2, 2001, 7:15 PM] foma Downtime
foma (www176) crashed under heavy load and had to be rebooted. Total
downtime was approximately 10 minutes.
[Apr 2, 2001, 3:14 PM] bemnet Downtime
bemnet (www158) crashed under heavy load, and had to be manually rebooted.
Total downtime was approximately 15 minutes.
[Apr 2, 2001, 2:23 PM] kodh Downtime
kodh (www90) crashed under heavy load, and had to be rebooted. Total
downtime was approximately 10 minutes.
[Mar 30, 2001, 10:28 PM] gwind Downtime
gwind (www74) crashed under heavy load, and has been returned to normal
service. Downtime was about 10 minutes.
[Mar 29, 2001, 7:37 PM] ungwe Maintenance Completed
We have completed the drive upgrade of ungwe (www48). It now has a 30GB
disk. Total downtime was bout 10 minutes.
[Mar 29, 2001, 1:42 AM] silme Maintenance Completed
We have completed the drive upgrade of silme (www64). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 28, 2001, 11:12 PM] anga Maintenance Completed
We have completed the drive upgrade of anga (www47). It now has a 30GB
disk. Final downtime was about 5 minutes.
[Mar 28, 2001, 10:57 PM] anga Downtime
anga (www47) crashed during the drive swap, and required a manual disk
check. Total downtime was about 45 minutes. We will now resume and complete
the drive swap, and anticipate a brief period of downtime again later
tonight to complete the swap.
[Mar 28, 2001, 7:50 PM] ungwe Planned Maintenance
We are about to begin a drive upgrade of ungwe (www48) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 28, 2001, 7:36 PM] silme Planned Maintenance
We are about to begin a drive upgrade of silme (www64) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 28, 2001, 7:24 PM] anga Planned Maintenance
We are about to begin a drive upgrade of anga (www47) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 28, 2001, 6:57 PM] aaze Maintenance Completed
We have completed the drive upgrade of aaze (www65). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 28, 2001, 5:21 PM] Web Site Maintenance
In response to early warning signs of drive trouble, we took our primary
Web server down briefly today in order to complete a drive swap. This
had no effect on customer traffic, DNS, or mail delivery, with the
exception that any customer relying on redirects from www.pair.com/username
would not be redirected during that time. We do not recommend relying
on that form of URL.
Our main Web site and signup system were briefly offline during the swap,
but we do not expect any further problems.
[Mar 28, 2001, 2:56 PM] Genuity Problems Resolved
Genuity has resolved their problem reaching our new connectivity in
Cleveland Ohio. Traffic is now flowing normally through our Genuity DS-3.
Some customers may experienced slightly improved performance as a result of
this changeover; we also have improved future expansion capacity.
[Mar 28, 2001, 2:46 PM] Genuity Maintenance Problems
Genuity is having trouble delivering some traffic from their network
through our new Cleveland-based connection. We are working with them to
resolve the problem; if it cannot be resolved shortly, we will be switching
the circuit back to Washington DC.
[Mar 28, 2001, 12:56 PM] Genuity Maintenance
There may be a brief interruption to our network service through Genuity
during the next hour; we are working with their NOC to move our circuit
from Washington DC to Cleveland Ohio, for more direct connectivity.
[Mar 27, 2001, 1:33 AM] ando Maintenance Completed
We have completed the drive upgrade of ando (www45). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 26, 2001, 11:05 PM] hyarmen Maintenance Completed
We have completed the drive upgrade of hyarmen (www66). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 26, 2001, 10:05 PM] aaze Planned Maintenance
We are about to begin a drive upgrade of aaze (www65) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 26, 2001, 10:02 PM] quesse Maintenance Completed
We have completed the drive upgrade of quesse (www44). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 26, 2001, 8:31 PM] ando Planned Maintenance
We are about to begin a drive upgrade of ando (www45) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 26, 2001, 7:40 PM] hyarmen Planned Maintenance
We are about to begin a drive upgrade of hyarmen (www66) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 26, 2001, 6:14 PM] quesse Planned Maintenance
We are about to begin a drive upgrade of quesse (www44) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 25, 2001, 5:06 PM] ceirt Downtime
After the second crash in two days, ceirt refused to come back online.
The problem was traced to a faulty hard drive cable. Upon swapping this
cable, the server is coming back online now. Total downtime was
approximately thirty minutes.
[Mar 25, 2001, 3:06 AM] yanta Maintenance Completed
We have completed the drive upgrade of yanta (www67).It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 25, 2001, 2:55 AM] tama Maintenance Completed
We have completed the drive upgrade of tama (www70). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 25, 2001, 2:28 AM] quesse Planned Maintenance
Due to a previously unforseen hardware issue, the drive swap of quesse
(www44) will be put off to a later date. We apologize for any
inconvenience.
[Mar 25, 2001, 1:46 AM] quesse Planned Maintenance
We are about to begin a drive upgrade of quesse (www44) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 25, 2001, 1:42 AM] parma Maintenance Completed
We have completed the drive upgrade of parma (www42). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 24, 2001, 10:49 PM] yanta Planned Maintenance
We are about to begin a drive upgrade of yanta (www67) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 24, 2001, 10:34 PM] parma Planned Maintenance
We are about to begin a drive upgrade of parma (www42) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 24, 2001, 10:21 PM] mildh Maintenance Completed
We have completed the drive upgrade of mildh (www87). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 24, 2001, 10:09 PM] tama Planned Maintenance
We are about to begin a drive upgrade of tama (www70) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 24, 2001, 10:00 PM] ceirt Maintenance Completed
We have completed the drive upgrade of ceirt (www39). It now has a 20GB
disk. We also took this opportunity to swap suspected bad RAM. Total
downtime was about 30 minutes.
[Mar 24, 2001, 9:34 PM] bellat Maintenance Completed
We have completed the drive upgrade of bellat (www101). It now has a 20GB
disk. Total downtime was about 10 minutes.
[Mar 24, 2001, 5:55 AM] thnad Downtime
thnad (www121) crashed under high load and had to be rebooted. Total
downtime was less than five minutes.
[Mar 22, 2001, 10:55 PM] ceirt Planned Maintenance
We are about to begin a drive upgrade of ceirt (www39) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 22, 2001, 10:37 PM] mildh Planned Maintenance
We are about to begin a drive upgrade of mildh (www87) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 22, 2001, 10:03 PM] scire Downtime
scire (www78) suffered a hardware failure, and had to be swapped to a new
chassis. Total downtime was about 20 minutes.
[Mar 22, 2001, 9:39 PM] bellat Planned Maintenance
We are about to begin a drive upgrade of bellat (www101) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 22, 2001, 2:58 PM] dhaame Downtime
dhaame (www77) was rebooted after it became unresponsive. After a file
systems check, it has returned to normal service again.
[Mar 21, 2001, 10:39 PM] gao Emergency Maintenance
gao (www111) was taken down for emergency maintenance to swap out a
suspected faulty RAM. Downtime was about 5 minutes, and the server
has been returned to normal service.
[Mar 21, 2001, 10:06 PM] gao Downtime
gao (www111) crashed due to heavy load. It required a manual system check
before being put back in service. Downtime was 20 minutes.
[Mar 20, 2001, 1:04 PM] flaid Downtime
flaid crashed under heavy load and had to be manually rebooted. Total
downtime was under 15 minutes.
[Mar 16, 2001, 12:00 PM] ciot Downtime
ciot, www161 crashed under heavy load, and was brought back
online after a manual reboot. Downtime was approximately
fifteen minutes.
[Mar 16, 2001, 2:02 AM] shen Maintenance Completed
We have completed the drive upgrade of shen (www88). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 14, 2001, 12:00 PM] ynilo Downtime
ynilo (www166) was rebooted when the system halted. After a disk and file
systems check, it has returned to normal service. Downtime was
approximately 10 minutes.
[Mar 13, 2001, 1:50 PM] foma Downtime
foma (www176) crashed under heavy load, and was brought back online.
Downtime was approximately ten minutes.
[Mar 13, 2001, 1:43 AM] bellat Planned Maintenance
Due to unforseen difficulties with the drive swap on bellat (www101), the
completion fo said drive swap will be postponed until further notice.
[Mar 13, 2001, 1:07 AM] tenshian Maintenance Completed
We have completed the drive upgrade of tenshian (www103). It now has a 30GB
disk. Total downtime was about 15 minutes.
[Mar 12, 2001, 11:39 PM] shen Planned Maintenance
We are about to begin a drive upgrade of shen (www88) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 12, 2001, 11:27 PM] vert Maintenance Completed
We have completed the drive upgrade of vert (www102). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 12, 2001, 9:38 PM] vert Planned Maintenance
We are about to begin a drive upgrade of vert (www102) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 12, 2001, 9:16 PM] cuzea Maintenance Completed
We have completed the drive upgrade of cuzea (www94). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 12, 2001, 8:57 PM] bellat Planned Maintenance
We are about to begin a drive upgrade of bellat (www101) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade
could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 12, 2001, 8:40 PM] tenshian Planned Maintenance
We are about to begin a drive upgrade of tenshian (www103)in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade
could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 12, 2001, 7:31 AM] umbar Downtime
umbar (www46) crashed under high load and had to be rebooted. Total
downtime approximately 7 minutes.
[Mar 11, 2001, 11:24 PM] dhabat Maintenance Completed
We have completed the drive upgrade of dhabat (www105). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 11, 2001, 11:12 PM] wredhor Maintenance Completed
We have completed the drive upgrade of wredhor (www76). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 11, 2001, 10:21 PM] cuzea Planned Maintenance
We are about to begin a drive upgrade of cuzea (www94) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 11, 2001, 10:02 PM] dhaame Maintenance Completed
We have completed the drive upgrade of dhaame (www77). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 11, 2001, 7:31 PM] dhabat Planned Maintenance
We are about to begin a drive upgrade of dhabat (www105) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 11, 2001, 7:19 PM] wredhor Planned Maintenance
We are about to begin a drive upgrade of wredhor (www76) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 11, 2001, 7:07 PM] otok Maintenance Completed
We have completed the drive upgrade of otok (www131). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 11, 2001, 6:56 PM] dhaame Planned Maintenance
We are about to begin a drive upgrade of dhaame (www77) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 11, 2001, 6:44 PM] lepen Maintenance Completed
We have completed the drive upgrade of lepen (www129). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 10, 2001, 2:58 AM] sether Downtime
system cleaning. Total downtime around 15 minutes.
[Mar 8, 2001, 9:42 PM] roomen Maintenance Completed
We have completed the drive upgrade of roomen (www61). It now has a 30GB
disk. Total downtime was about 10 minutes.
[Mar 8, 2001, 7:30 PM] otok Planned Maintenance
We are about to begin a drive upgrade of otok (www131) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 8, 2001, 7:20 PM] lepen Planned Maintenance
We are about to begin a drive upgrade of lepen (www129) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 8, 2001, 6:53 PM] sowilu Maintenance Completed
We have completed the drive upgrade of sowilu (www145). We also took this
opportunity to add more RAM to this server. It now has 256Mb of RAM and a
30Gb disk. Total downtime for this upgrade was about 10 minutes.
[Mar 8, 2001, 6:39 PM] roomen Planned Maintenance
We are about to begin a drive upgrade on roomen (www61) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 8, 2001, 6:19 PM] gao Maintenance Completed
We have completed the drive swap of gao (www111). It now has 30Gb of total
disk space. Final downtime was about 10 minutes.
[Mar 7, 2001, 11:58 PM] anga Downtime
anga crashed under high load and had to be rebooted. Downtime was less that
15 minutes.
[Mar 7, 2001, 10:56 PM] ifin Maintenance Completed
We have completed the drive upgrade of ifin (www36). It now has a 30Gb
disk. Total downtime was about 10 minutes.
[Mar 7, 2001, 9:59 PM] gao Emergency Maintenance
We were able to salvage the primary drive of gao (www111). All user content
is now back online, and the server is functioning properly.
We were not able, however, to salvage the new drive we were in the process
of swapping to. A new drive swap will begin shortly, and could take another
24 hours to completed.
We have also moved gao to a new chassis in the process of this emergency
downtime. It is now a Pentium III 866 MHz with 256Mb of RAM. This has
helped to improve performance on this server.
We apologize to the customers affected by this downtime.
[Mar 7, 2001, 9:07 PM] gao Emergency Maintenance
gao (www111) suffered a primary drive failure before the planned drive swap
completed successfully. We are working on recovering the data and bringing the
server back into service. Expected completion is within the next 3 hours.
More details will be posted as they become available.
[Mar 7, 2001, 6:16 PM] ifin Planned Maintenance
We are about to begin a drive upgrade on ifin (www36) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 7, 2001, 1:50 PM] psi Downtime
psi (www14) rebooted, and was down during a file systems cleanup. It has
returned to normal service in under 10 minutes.
[Mar 7, 2001, 8:50 AM] yanta Downtime
yanta crashed under high load and had to be rebooted. Downtime was less
than 15 minutes.
[Mar 6, 2001, 10:54 PM] gwind Maintenance Completed
We have completed the drive swap on gwind (www74). It now has a 30GB
disk. Total downtime was around 10 minutes.
[Mar 6, 2001, 7:37 PM] gwind Planned Maintenance
We are about to begin a drive upgrade on gwind (www74) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 6, 2001, 6:39 PM] gao Planned Maintenance
We are about to begin a drive upgrade on gao (www111) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 6, 2001, 5:47 PM] sowilu Planned Maintenance
We are about to begin a drive upgrade on sowilu (www145) in order to
improve storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 4, 2001, 10:21 PM] jarre Downtime
jarre crashed under high load and had to be rebooted. Downtime was less
than 15 minutes.
[Mar 4, 2001, 8:30 PM] yuzz Maintenance Completed
We have completed the drive swap on yuzz (www117). It now has a 30GB disk.
Total downtime was around 10 minutes.
[Mar 3, 2001, 10:04 PM] yuzz Planned Maintenance
We are about to begin a drive upgrade on yuzz (www117) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 3, 2001, 9:28 PM] db16 RAM swap
We will be taking db16 down briefly in order to swap out suspected bad RAM.
We believe this will help to eliminate the issues this server is
experiencing. Downtime for this maintenance will be about 10 minutes.
[Mar 3, 2001, 6:24 PM] huath, reit, khurla, naam, glikk, zatz, flunn, fehu, uruz Planned Maintenance
We will be taking down the following servers between 6:30 PM and 10:00 PM
in order to upgrade their ethernet cards:
huath (www19)
reit (www108)
khurla (www112)
naam (www114)
glikk (www119)
zatz (www112)
flunn (www123)
fehu (www134)
uruz (www135)
Total downtime for each server will be about 10 minutes.
[Mar 3, 2001, 1:19 AM] scire Downtime
downtime under 10 minutes.
[Mar 2, 2001, 8:34 AM] anca Log Problem
Due to a data corruption error, logs for March 1 have been truncated on
anca (www53). If this recurs, we will swap RAM in the system, as that is
the most likely culprit. Truncated logs are being generated now.
[Mar 1, 2001, 9:10 PM] maor Maintenance Completed
We have completed the drive upgrade of maor (www113). It now has a 30GB
hard drive. Total downtime was about 10 minutes.
[Mar 1, 2001, 8:50 PM] lissa Maintenance Completed
We have completed the drive upgrade of lissa (www72). It now has a 30GB
hard drive. Total downtime was about 10 minutes.
[Mar 1, 2001, 8:27 PM] kanat Maintenance Completed
We have completed the chassis swap of kanat (www128). It is now a Pentium
III 866 MHz with 256Mb of RAM. Total downtime for this maintenance was
about 10 minutes.
[Mar 1, 2001, 8:05 PM] kanat Planned Maintenance
In order to replace a suspected dying power supply, we will be taking kanat
(www128) down briefly in order to swap it to a new chassis. Total downtime
for this maintenance should be about 10 minutes. We will post an additional
notice when the maintenance has been completed.
[Mar 1, 2001, 6:40 PM] maor Planned Maintenance
We are about to begin a drive upgrade on maor (www113) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 1, 2001, 6:35 PM] kanat Maintenance Completed
We have completed the drive upgrade of kanat (www128). It now has a 30GB
hard drive. Total downtime was about 10 minutes.
[Mar 1, 2001, 5:56 PM] lissa Planned Maintenance
We are about to begin a drive upgrade on lissa (www72) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Mar 1, 2001, 7:05 AM] emancholl downtime
emancholl crashed under high load and had to be rebooted. Downtime was about 5 minutes.
[Mar 1, 2001, 12:22 AM] neled Upgrade Completed
We have completed the drive upgrade of neled (www127). It now has a 30GB
hard drive. Total downtime was about 10 minutes.
[Feb 28, 2001, 10:18 PM] kanat Upgrade
We are about to begin a drive upgrade on kanat (www128) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 28, 2001, 9:55 PM] thnad Upgrade Completed
We have completed the drive upgrade of thnad (www121). It now has a 30GB
hard drive. Total downtime was about 10 minutes.
[Feb 28, 2001, 7:17 PM] neled Upgrade
We are about to begin a drive upgrade on neled (www127) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 28, 2001, 7:07 PM] mini Upgrade Completed
We have completed the drive upgrade of mini (www126). It now has a 30GB
hard drive. Final downtime was around 5 minutes.
[Feb 28, 2001, 6:11 PM] thnad Upgrade
We are about to begin a drive upgrade of thnad (www121) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 26, 2001, 8:42 PM] pyyl Maintenance Completed
We have completed the emergency drive maintenance of pyyl (www97). Total
downtime was about 10 minutes.
[Feb 26, 2001, 8:27 PM] flure Upgrade Completed
We have completed the drive upgrade of flure (www75). It now has a 30GB
hard disk. Total downtime was about 10 minutes.
[Feb 26, 2001, 7:52 PM] nish Upgrade Completed
We have completed the drive and system upgrade of nish (www92). It was
originally swapped to a chassis that turned out to be bad. This caused more
downtime than usual. It is now an Athalon Thunderbird 1000 MHz with 256MB
of RAM and a 30GB disk. Total downtime was about 40 minutes.
[Feb 26, 2001, 7:50 PM] mini Downtime
Due to an error with it's ethernet card, mini (www126) was down for about
40 minutes. It is now back up, and we are about to continue with it's
upgrade.
[Feb 26, 2001, 6:34 PM] mini Upgrade
We are about to begin a drive upgrade of mini (www126) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 26, 2001, 6:30 PM] vroo Upgrade Completed
We have completed the drive upgrade of vroo (www125). It now has a 30GB
hard disk. Total downtime was about 10 minutes.
[Feb 26, 2001, 12:14 PM] pyyl emergency maintenance
We will be taking down pyyl (www97) for emergency hard drive maintanance.
There will be two brief periods of downtime, each approximately five
minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to
48 hours to complete. We will post an additional notice when the upgrade
has been completed.
[Feb 26, 2001, 10:37 AM] aveu Downtime
aveu (www192) crashed under a heavy load and was rebooted. The downtime
was under 10 minutes.
[Feb 25, 2001, 10:41 PM] flure Upgrade
We are about to begin a drive upgrade of flure (www75) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 25, 2001, 10:37 PM] edad Upgrade Completed
We have completed the hard drive upgrade of edad. It now has a 30GB disk.
Total downtime was about 10 minutes.
[Feb 25, 2001, 9:27 PM] vroo Upgrade
We are about to begin a drive upgrade of vroo (www125) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 25, 2001, 9:24 PM] flaid Upgrade Completed
We have completed the drive and system upgrade of flaid (www91). It is now
an Athalon Thunderbird 1000 MHz with 256Mb of RAM and a 16GB hard drive.
Total downtime was about 10 minutes.
[Feb 25, 2001, 7:43 PM] edad Upgrade
We are about to begin a drive upgrade on edad (www31) in order to improve
storage. There will be two brief periods of downtime, each approximately
five minutes, at the beginning and end of the maintenance. The server will
remain online at all other times. The entire upgrade could take up to 24
hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 25, 2001, 7:25 PM] vuae Upgrade Completed
We have completed the drive and system upgrade of vuae (www93). It is now
an Athalon Thunderbird 1000 MHz with 256Mb of RAM and a 30 GB hard drive.
Total downtime was about 10 minutes.
[Feb 25, 2001, 6:55 PM] nish Upgrade
We are about to begin a drive and system upgrade of nish (www92) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 25, 2001, 6:37 PM] flaid Upgrade
We are about to begin a drive and system upgrade of flaid (www91) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 25, 2001, 6:35 PM] sether Upgrade Completed
We have completed the drive and system upgrade of sether (www95). It is
now an Athalon Thunderbird 1000 MHz with 256Mb of RAM and a 30 GB hard
drive. Total downtime was about 20 minutes.
[Feb 25, 2001, 12:45 PM] db16 Downtime
db16 crashed under heavy load, and had to be rebooted. Total downtime was
around 10 minutes.
[Feb 25, 2001, 12:42 PM] neled Downtime
neled (www127) crashed under heavy load, and had to have it's filesystems
cleaned. Downtime, 17 minutes.
[Feb 24, 2001, 2:44 PM] fearn Downtime
fearn (www40) crashed under heavy load, and was brought back online with 11
minutes downtime.
[Feb 23, 2001, 11:45 PM] voot Downtime
voot (www154) crashed under high load and had to be rebooted. Total
downtime; 15 minutes.
[Feb 23, 2001, 10:01 AM] dotya reboot
dotya (www170) was rebooted to clear up a high load condition. Downtime
was approximately two minutes.
[Feb 23, 2001, 8:36 AM] diaf Downtime
diaf (www215) crashed under load and was brought back online after
approximately 5 minutes of downtime.
[Feb 22, 2001, 11:04 PM] derba Upgrade Completed
We have completed the drive and system upgrade of derba (www104). It is now
a Pentium III 866 MHz with 256 MB of RAM and a 30GB hard drive. Total
downtime was under five minutes.
[Feb 22, 2001, 10:26 PM] vuae Upgrade
We are about to begin a drive and system upgrade of vuae (www93) in order
to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 10:22 PM] zirx Upgrade Completed
We have completed the drive and system upgrade of zirx (www106). It is now
a Pentium III 866 MHz with 256 MB of RAM and a 30GB hard drive. Total
downtime was under five minutes.
[Feb 22, 2001, 9:34 PM] thesel Upgrade Completed
We have completed the system upgrade of thesel (www99). It is now a Pentium
III 866 MHz with 256 MB of RAM. Total downtime was under five minutes.
[Feb 22, 2001, 9:13 PM] thesel Upgrade
We are about to begin a system upgrade on thesel (www99) in order to
improve performance. There will be a brief period of downtime, about five
minutes. A notice will be posted once the upgrade is complete.
[Feb 22, 2001, 8:52 PM] sether Upgrade
We are about to begin a drive and system upgrade on sether (www95) in order
to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 8:49 PM] vilya Upgrade Completed
We have completed the drive and system upgrade of vilya (www60). It is now
a Pentium III 866 MHz, with 256MB RAM and a 30.7GB disk. Total downtime was
under 10 minutes.
[Feb 22, 2001, 7:16 PM] derba Upgrade
We are about to begin a drive and system upgrade on derba (www104) in order
to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 7:12 PM] bemnet Upgrade Completed
We have completed the drive upgrade of bemnet (www158). It now has a 30GB
hard drive. We also added more RAM to this server, bringing it to 256MB.
Total downtime was under 10 minutes.
[Feb 22, 2001, 6:37 PM] zirx Upgrade
We are about to begin a drive and system upgrade on zirx (www106) in order
to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 6:32 PM] aster Upgrade Completed
We have completed the drive and system upgrade of aster (www96). It is now
a Pentium III 866 MHz with 256 MB of RAM and a 30 GB hard drive. Total
downtime was under 10 minutes.
[Feb 22, 2001, 9:32 AM] db16 Maintenance
db16 will be taken down briefly to begin the installation of a new
hard drive. When the maintenance is complete, there will be another brief
period of downtime. The server will remain online at all other times
during the upgrade.
[Feb 22, 2001, 4:36 AM] vilya Upgrade
We are about to begin a drive and system upgrade on vilya (www60)
in order to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 4:33 AM] ailm Upgrade Completed
We have completed the upgrade of ailm (www28). It is now a Pentium
III 866 MHz with 256MB of RAM and a 30.5GB hard drive. Total downtime was
under 5 minutes.
[Feb 22, 2001, 3:29 AM] aster Upgrade
We are about to begin a drive and system upgrade on aster (www96)
in order to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 3:27 AM] gnaaste Upgrade Completed
We have completed the upgrade on gnaaste (www79). It is now a
Pentium III 866 MHz with 256MB of RAM and a 20GB hard drive. Total downtime
was under 10 minutes.
[Feb 22, 2001, 2:29 AM] ailm Upgrade
We are about to begin a drive and system upgrade on ailm (www28)
in order to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times.
The entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 22, 2001, 2:24 AM] ydhu Upgrade Completed
We have completed the upgrade on ydhu (www83). It is now a Pentium
III 866 MHz with 256MB of RAM and a 16GB hard drive.
[Feb 22, 2001, 12:17 AM] enda Upgrade Completed
We have completed the upgrade of enda (www80). Due to a problem
during the upgrade process, downtime was approximately 30 minutes. It is
now a Pentium III 866 MHz with 256Mb of RAM.
[Feb 21, 2001, 11:22 PM] enda Upgrade
We are about to begin a system upgrade on enda (www80) in order to
improve performance. There will be a brief period of downtime,
approximately ten minutes.
[Feb 21, 2001, 11:09 PM] ilwe Upgrade Completed
We have completed the upgrade of ilwe (www81). It is now a Pentium
III 866 MHz with 256 Mb of RAM. Downtime was about 10 minutes.
[Feb 21, 2001, 10:36 PM] ilwe Upgrade
We are about to begin a system upgrade on ilwe (www81) in order to
improve performance. There will be a brief period of downtime,
approximately ten minutes.
[Feb 21, 2001, 9:52 PM] gnaaste Upgrade
We are about to begin a drive and system upgrade on gnaaste (www79)
in order to improve performance and storage. There will be two brief
periods of downtime, each approximately five minutes, at the beginning and
end of the maintenance. The server will remain online at all other times. The
entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 21, 2001, 9:20 PM] ydhu Upgrade
We are about to begin a drive and system upgrade on ydhu (www83) in
order to improve performance and storage. There will be two brief periods
of downtime, each approximately five minutes, at the beginning and end of
the maintenance. The server will remain online at all other times. The
entire upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 21, 2001, 9:14 PM] ngatwo Upgrade Completed
We have completed the drive upgrade on ngatwo (www115). It now has
a 30 GB hard drive. Total downtime was under 10 minutes.
[Feb 21, 2001, 6:24 PM] bemnet Upgrade
We are about to begin a drive upgrade on bemnet (www158) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 21, 2001, 5:53 PM] ngatwo Upgrade
We are about to begin a drive upgrade on ngatwo (www115) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete.
We will post an additional notice when the upgrade has been completed.
[Feb 21, 2001, 5:21 PM] RAM upgrades
The following servers will be taken down for RAM upgrades between 6:00 PM
and 10:00 PM EST tonight:
uilen (www35)
kirlian (www110)
wunjo (www140)
naudhiz (www141)
perthro (www143)
tiwaz (www146)
berkano (www146)
haebrath (www156)
uumor (www159)
Downtime should be under 10 minutes for each server.
[Feb 21, 2001, 12:40 PM] db16 maintenance completed
The hardware swap for db16 has been completed. Downtime was
approximately five minutes.
[Feb 21, 2001, 11:32 AM] kodh downtime
kodh (www90) crashed under high load, and was brought back online.
Downtime was less than 10 minutes.
[Feb 21, 2001, 11:19 AM] db16 emergency maintenance planned
Due to some problems with its hardware, we will be swapping db16
into a new chassis around approximately 12:30 PM Eastern. Downtime should
be under ten minutes.
[Feb 21, 2001, 10:03 AM] kodh Downtime
kodh (www90) crashed under heavy load, and had to be manually rebooted after a thorough disk check. Total downtime was approximately 15 minutes.
[Feb 20, 2001, 5:18 AM] elhaz downtime
elahz crashed under high load and had to be rebooted. Downtime was under 10 minutes.
[Feb 19, 2001, 10:44 AM] kayan reboot
kayan (www133) was rebooted to clear up a high load condition.
Downtime was approximately two minutes.
[Feb 12, 2001, 4:32 PM] Billing Invoices
On Sunday, February 11th, overdue notices were sent out that mistakenly
omitted the total amount that was overdue. In some cases, notices were
sent out for accounts that were not overdue. If you received an overdue
notice from the Billing Department, please contact billing@pair.com for
clarification or more information. Please note that invoices generated
during the month of February 2001 are due on the 28th.
[Feb 12, 2001, 2:20 PM] rult Downtime
rult (www162) crashed under heavy load and had to be manually rebooted. Total d
owntime was approximately 10 minutes.
[Feb 9, 2001, 6:20 PM] chi downtime
chi (www14) crashed under heavy load and had to be manually
rebooted. Downtime was about 10 minutes.
[Feb 9, 2001, 6:16 PM] jarre Upgrade Completed
The RAM upgrade of jarre (www172) has been completed. It now has
512MB of RAM. Downtime was under 5 minutes.
[Feb 9, 2001, 5:54 PM] jarre Upgrade
We will be taking jarre (www172) down after 6pm EST to complete
the RAM upgrade, postponed from yesterday. Downtime should be under 5
minutes, and a notice will be posted afterward.
[Feb 8, 2001, 8:06 PM] tinco Maintenance
We will be taking tinco (www41) down breifly to resituate it's
hard drive. Downtime should be about 10 minutes.
[Feb 8, 2001, 8:01 PM] jarre Maintenance Postponed
Due to issues with the RAM in jarre (www172), the upgrade for this
server has been postponed. There was a period of approximately fifteen
minutes of downtime while the server was reverted to its previous
configuration.
[Feb 8, 2001, 3:34 PM] bemnet Downtime
bemnet (www158) crashed under heavy load and had to be manually rebooted. Total downtime was approximately 10 minutes.
[Feb 8, 2001, 1:41 PM] fearn Downtime
fearn (www40) crashed under heavy load and had to be rebooted. Total downtime was approximately 10 minutes.
[Feb 8, 2001, 11:29 AM] jarre Maintenance
We have scheduled a RAM upgrade for jarre (www172), to be
completed after 6:00 PM Easterm time. After the upgrade, jarre
will have 512 MB RAM.
[Feb 7, 2001, 9:27 PM] dotya Downtime
dotya (www170) crashed under heavy load and required a manual restart.
Downtime was 10 minutes.
[Feb 7, 2001, 9:04 PM] theta Upgrade
We have added more RAM to theta (www4). It now has 384 MB of RAM.
Downtime was under 5 minutes.
[Feb 7, 2001, 8:42 PM] jera Upgrade
We have added more RAM to jera (www42). It now has 256 MB of RAM.
Downtime was under 5 minutes.
[Feb 7, 2001, 8:30 PM] scire Upgrade Completed
We have completed the hardware upgrade of scire (www78). It is now
an Athlon Thunderbird 1000 MHz with 256 MB of RAM.
[Feb 7, 2001, 8:13 PM] nuumen Upgrade Complete
We have completed the hardware upgrade of nuumen (www55). It is
now a Pentuim II 866 MHz with 256MB of RAM.
[Feb 7, 2001, 7:49 PM] coll Upgrade Completed
We have completed the hardware upgrade on coll (www22). Is is now
a Pentium II 866 MHz with 256Mb of RAM.
[Feb 7, 2001, 7:32 PM] xrra Upgrade Completed
We have completed the hardware upgrade of xrra (www116). Is is now
a Pentium III 866 MHz with 256 MB of RAM.
[Feb 7, 2001, 6:53 PM] Server Maintenance
We are about to perform hardware upgrades on the following servers:
xrra (www16)
coll (www22)
nuumen (www55)
scire (www78)
Downtime should be five minutes or less per server. An additional notice
will be posted after each server is upgraded.
We will also be adding RAM to theta (www4) and jera
(www42) to improve their performance. Downtime should be under five minutes
for this maintenance.
[Feb 7, 2001, 4:40 PM] omega downtime
omega (www16) crashed under heavy load and was rebooted. Dowtime was under
10 minutes.
[Feb 7, 2001, 11:27 AM] xrra Maintenance
The FreeBSD -STABLE upgrade has been delayed for xrra (www116);
it will require a hardware upgrade before it meets our minimum requirements
for running -STABLE. The hardware upgrade will be completed this evening
after 6pm Eastern time. The -STABLE upgrade has been rescheduled for
Thursday, February 8.
[Feb 7, 2001, 11:24 AM] elhaz Maintenance Completed
The emergency upgrade of elhaz (www144) has been completed. The
server is now an Athlon Thunderbird at 1000 MHz, with 384MB SDRAM.
[Feb 7, 2001, 10:42 AM] elhaz Maintenance
Based on the continued high load of elhaz (www144), we have
promoted its scheduled hardware upgrade to emergency status. The upgrade
will be completed within the next 15 minutes.
[Feb 7, 2001, 10:07 AM] elhaz downtime
Due to high load from a user script, elhaz (www144) was
unresponsive for close to 5 minutes. The server has been recovered without
need for a reboot, and is operating normally again.
[Feb 7, 2001, 9:15 AM] elhaz Downtime
elhaz (www144) crashed under heavy load and was brought back
online promptly.
We have scheduled an upgrade for this server, to take place after 6pm
Eastern time today. The upgrade will include a faster processor and three
times as much RAM. This should significantly alleviate load problems.
[Feb 7, 2001, 6:37 AM] formmail.pl Fix
A fix for problems with formmail.pl encountered on some 4.1.1-STABLE
servers has been posted along with other minor updates to the code.
Any customer who continues to encounter trouble with the system CGI
formmail.pl script, please contact us.
[Feb 6, 2001, 2:16 AM] voot Downtime
voot (www154) was down this morning for approximately 10 minutes. It has
since returned to normal operation.
[Feb 5, 2001, 11:28 AM] naudhiz Downtime
naudhiz (www141) crashed under heavy load and was rebooted. The downtime
was under 5 minutes.
[Feb 3, 2001, 8:03 PM] bhoth Maintenance
Because we have observed several recent crashes on bhoth
without reasonable explanation, we will be taking the server down
briefly for a RAM swap; downtime will be less than five minutes, and
the maintenance will be completed before 11pm Eastern time.
[Feb 3, 2001, 9:59 AM] Network Problems
We are currently working to trace and identify a massive packet flood which
affected our network operations between 7:50am and 9am Eastern time today.
It is not yet clear if this stemmed from an external attack or an internal
problem. We will post further details as they become available.
[Feb 2, 2001, 8:28 AM] aedde Upgrade Complete
We have completed the upgrade of aedde (www84). It is now a
Pentium III 866 Mhz, with 256 Mb of RAM and 30 Gb of total hard disk space.
Total downtime was 10 minutes.
[Feb 2, 2001, 3:07 AM] aedde Upgrade
We are about to begin a drive and system upgrade on aedde (www84)
in order to improve performance and storage. There will be two brief periods
of downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Feb 1, 2001, 11:33 PM] aedde upgrade postponed
The previously announced upgrade for aedde (www84) has been
postponed to a future date.
[Feb 1, 2001, 11:14 PM] yuzz upgrade completed
The upgrade of yuzz (www117) has been completed.
The server is now a Pentium III 866 MHz, with 256 MB of RAM.
Downtime was under five minutes.
[Feb 1, 2001, 10:56 PM] paat upgrade completed
The upgrade of paat (www100) has been completed.
The server is now a Pentium III 866 MHz, with 256 MB of RAM.
Downtime was under five minutes.
[Feb 1, 2001, 10:39 PM] falku upgrade completed
The upgrade of falku (www98) has been completed.
The server is now a Pentium III 866 MHz, with 256 MB of RAM.
Downtime was under five minutes.
[Feb 1, 2001, 10:16 PM] ulwar upgrade completed
The upgrade of ulwar (www82) has been completed.
The server is now a Pentium III 866 MHz, with 256 MB of RAM.
Downtime was under five minutes.
[Feb 1, 2001, 4:47 PM] Miva Performance Issues
Some customers have observed performance problems with the Miva Merchant
software included with our Commerce Miva account. We have recently traced
these problems to bugs in the Miva Empresa engine which we believe are
not present in the version for FreeBSD 3.x. We are in the process of
upgrading all user servers to FreeBSD 4.1.1-STABLE, but we have found that
a different bug in the implementation of passwords in newer versions of Miva
will prevent customer sites from working properly after such an upgrade.
For the past ten days, we have attempted to work with Miva to resolve this
bug in their software. Ideally, they would build a fixed version of Miva
for the FreeBSD 4.x platform; they are only supporting FreeBSD 3.x, which
is a dead-end track with limited security fixes and numerous other problems.
Unfortunately, after ten days, they have only offered us a method to reset
user passwords, which besides disrupting customer sites, does nothing to
assure us that there will not be future problems in their password code.
We are disheartened, but are working on our own approaches to the problem.
We will decide on the appropriate solution by Wednesday, February 7. Please
note that there is no intent to disrupt our customers using Miva, nor do we
intend to discontinue our partnership with Miva. We are simply dissatisfied
with their response to date.
We expect that in the worst case, customers will be issued new passwords
immediately upon upgrade. We will continue to press for a version of Miva
which supports our needs, however.
[Feb 1, 2001, 12:07 PM] Server maintenance
We are expecting to perform hardware upgrades on the following servers
tonight, approximately around 10:00 PM and 12:00 AM Eastern:
ulwar (www82)
aedde (www84)
falku (www98)
paat (www100)
yuzz (www117)
Downtime should be five minutes or less per server. An additional notice
will be posted after each server is upgraded.
[Jan 31, 2001, 7:53 PM] bhoth Downtime
bhoth (www163) was down this evening for approximately 5 minutes. It has
since returned to normal operation.
[Jan 31, 2001, 4:28 PM] jera Downtime
jera (www142) crashed under heavy load and had to be manually rebooted. Total downtime was approximately 15 minutes.
[Jan 31, 2001, 4:07 PM] zais Maintenance Completed
We have completed the drive swap on zais, with no further
difficulties encountered. The server is now running stable as a Pentium
III at 866 MHz, with 256MB SDRAM and a 30GB drive. We will continue to
monitor it closely for any further problems.
Any customer who observes unusual behavior or configuration problems from
this server, please contact us at urgent@pair.com immediately.
We believe that these swaps will eliminate all further instability.
[Jan 31, 2001, 11:30 AM] gebo Downtime
gebo (www139) crashed under heavy load, and had to be manually rebooted. Total downtime was approximately 15 minutes.
[Jan 31, 2001, 9:30 AM] zais Maintenance
During the resumed swap of zais' original hard drive this morning,
the new destination drive apparently failed. We are not yet certain if
this was caused by a genuine drive failure, or a possible problem with
the old hard drive which is affecting system stability.
We have attempted to work around that instability, and will be resuming
the swap using another new drive. The server will be down briefly once
more as this swap is begun.
All of our new drives undergo extensive
burn-in testing, but failures are always a possibility. In the event of
worst possible disaster, we have verified that backups of this server are
less than 24 hours old.
Please accept our apologies for these continued problems; we are striving
to establish the same level of performance and reliability that most of
our servers already achieve.
[Jan 31, 2001, 8:30 AM] zais Maintenance
zais (www89) will be taken down briefly to be swapped into a new
chassis, and the drive swap process will be resumed.
[Jan 31, 2001, 7:05 AM] zais Maintenance
We have concluded that the unstable behavior exhibited by zais
is due either to failure of its older hard drive, or some problem related
to its new motherboard or CPU. This morning we will be replacing the
zais chassis and proceeding with the drive swap. As soon as the
drive swap is finished, the server will be taken down briefly to complete
the swap.
Because the server is unstable, this work will be conducted on an emergency
basis outside of our normal maintenance window. Updates will be posted
here as new information becomes available.
[Jan 31, 2001, 5:10 AM] zais downtime
zais (www89) crashed under high load and had to have it's file system
manually cleaned. Downtime, 20 minutes.
[Jan 30, 2001, 11:02 PM] zais Upgrade
We are about to begin a drive swap on zais (www89), to improve
performance and storage. There will be two brief periods of downtime, at
the beginning and end of the maintenance. The server will remain online at
all other times.
We will also take this opportunity to swap the RAM on zais. We
believe this end the unstable behavior this server has been experiencing.
[Jan 30, 2001, 10:57 PM] fearn Upgrade Complete
We have completed a drive and system upgrade on fearn (www40). It
is now an Athalon Thunderbird 1000 MHz with 256 MB of RAM, and 30 GB of
total disk space. Downtime was 10 minutes.
[Jan 30, 2001, 7:30 PM] zais Maintenance
We have identified unstable behavior on zais; there have been
three unnecessary system crashes since the system was upgraded on January
2nd. We will be swapping the RAM on this server within the next twelve
hours. A drive swap will also be initiated within the next three days.
We have confirmed that backups are up-to-date for this server.
Users running php3 scripts may have encountered problems in the last
several days, which were caused by a corrupted php3.cgi binary. This
has been corrected.
[Jan 30, 2001, 3:25 PM] zais Downtime
zaid (www89) crashed under heavy load and had to be manually rebooted after a thorough disk check. Total downtime was approximately 20 minutes.
[Jan 30, 2001, 11:05 AM] kodh Downtime
kodh (www90) crashed under heavy load and had to be manually rebooted. Total downtime was approximately 10 minutes.
[Jan 29, 2001, 9:09 PM] ur Update
We have discovered that a small number of recently-updated user files
were not restored from backup on ur.pair.com today. These files are
now being restored, and should be back to normal within the next two hours.
[Jan 29, 2001, 7:09 PM] kodh Downtime
kodh (www90) crashed under heavy load. The server required a manual file
check before being brought back online. Downtime was 25 minutes.
[Jan 29, 2001, 4:52 PM] ur recovery
The recovery of user files on ur (www30) has been completed. All
sites should again be functional. We are now completing this recovery
with an update of incremental backups, which will bring this server up to
date as of this morning. If there are any problems observed with this
server, please let us know at support@pair.com.
[Jan 29, 2001, 3:24 PM] ruis maintenance completed
We have finished the hard drive swap on ruis (www27). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 29, 2001, 3:15 PM] ur recovery
ur (www30) is currently undergoing a full systems recovery from
our backup systems. The old drive in this server, which ironicly was being
replaced by today's maintenance, unrecoverably failed. Our most recent
incremental backup was run on this server just this morning, so loss of
user data will be minimal. Sites hosted on ur may be inaccessable for up to
another hour while files are being recovered. We will post another notice
when this process has been completed.
[Jan 29, 2001, 2:35 PM] ur sites downtime
While beginning the maintenance on ur (www30), the old drive has
reached a point of unuseability. We are working to restore service to all
sites on this server as soon as possible. We will post an update when more
information is available.
[Jan 29, 2001, 2:00 PM] ur maintenance
We are about to begin a drive upgrade on ur (www30) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 29, 2001, 12:55 PM] ruis maintenance
We are about to begin a drive upgrade on ruis (www27) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 29, 2001, 12:43 PM] Nameserver Updates
Beginning this weekend and continuing throughout today, we have been
deploying updated nameserver code to all servers. There will be no
interruption of nameservice for most clients; however, the updated software
is less tolerant of certain syntax, and as a result, a few domains may go
out of service.
We are working to eliminate this problem in general. If you notice any
problems with resolving your domain name, please contact us at
urgent@pair.com today. This is a critical security update that we cannot
presently avoid.
[Jan 29, 2001, 12:25 PM] fearn maintenance
We are about to begin a drive upgrade on fearn (www40) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 29, 2001, 11:41 AM] mhoh maintenance completed
We have finished the hardware upgrade on mhoh (www191). The
server is now a Pentium III 866 MHz server, with 256 MB of RAM. Downtime
was under five minutes.
[Jan 29, 2001, 11:13 AM] mhoh maintenance
We are about to begin a hardware upgrade on mhoh (www191) in order
to relieve a minor problematic condition. Downtime should be less than
five minutes.
[Jan 28, 2001, 6:01 PM] vsael Downtime
vsael (www173) was down for approximately 10 minutes. It has since returned
to normal operation.
[Jan 28, 2001, 9:05 AM] omega Downtime
omega (www16) crashed and was rebooted. A manual disk check was performed. Total downtime was approximately 10 minutes.
[Jan 28, 2001, 12:00 AM] psi Upgrade
We have completed a hardware upgrade on psi (www15). It has been
upgraded to an Athlon Thunderbird 1000 MHz with 256 MB of RAM. Downtime i
was about 5 minutes.
[Jan 27, 2001, 11:59 PM] chi Upgrade
We have completed a hardware upgrade on chi (www14). It has been
upgraded to an Athlon Thunderbird 1000 MHz with 256 MB of RAM. Downtime was
about 5 minutes.
[Jan 27, 2001, 11:22 PM] upsilon Upgrade
We have performed a hardware upgrade on upsilon (www13). It has
been upgraded to an Athlon Thunderbird 1000 MHz with 256 MB of RAM.
Downtime was about 10 minutes.
[Jan 26, 2001, 4:53 PM] jera Downtime
jera (www142) crashed under heavy load, and was automatically rebooted. Total downtime was under 10 minutes.
[Jan 26, 2001, 1:45 PM] SAVVIS Congestion
We are observing significant congestion and packet loss on our
SAVVIS circuit, beginning sometime yesterday and worsening today.
We will be shifting traffic towards other providers in order to alleviate
this condition.
[Jan 25, 2001, 11:37 PM] omicron Downtime
omicron (www9) down this evening following what was essentially a denial
of service attack from a major search engine service. The search engine
spider issued more than 40 hits/second to two sites hosted on omicron,
disregarding the fact that both sites were generating content dynamically
via CGI. The resulting load average was over 400. We have blocked the
search spider from those sites, and will be working with that service to
identify why their spider has been so ill-behaved.
[Jan 25, 2001, 8:05 AM] omicron Downtime
omicron (www9) crashed and was brought back online
after a manual reboot.
[Jan 25, 2001, 5:07 AM] omicron downtime
omicron (www9) suffered from severly high load and had to be rebooted.
Downtime close to 20 minutes.
[Jan 25, 2001, 12:45 AM] omicron downtime
omicron (www9) crashed under high load and had to be rebooted. Total
downtime of about 20 minutes.
[Jan 24, 2001, 4:33 PM] wawrra Upgraded
During the FreeBSD -STABLE upgrade of wawrra (www109) today,
the server was also upgraded to an Intel Pentium III at 866 MHz, with
256MB SDRAM.
[Jan 23, 2001, 9:16 PM] kodh Maintenance
We have added more RAM to kodh (www90). It now has 384Mb of RAM.
Total downtown for this upgrade was under 5 minutes.
[Jan 23, 2001, 9:14 PM] gort Upgrade Complete
We have completed the drive upgrade of gort (www24). It now has a
total of 17GB of hard drive space. Total downtime was under 10 minutes.
[Jan 23, 2001, 8:01 PM] ngetal Upgrade Completed
We have completed the drive upgrade on ngetal (www25). It now has
20GB of total disk space.
[Jan 23, 2001, 7:42 PM] kirlian Upgrade Complete
We have completed the drive upgrade of kirlian (www110). It now has
15 Gb of total disk space.
[Jan 23, 2001, 3:39 PM] ngetal Upgrade
We are about to begin a drive swap on ngetal (www25). There will be
two brief periods of downtime, each approximately five minutes, at the
beginning and end of the maintenance. The server will remain online at all
other times. The entire upgrade could take up to 24 hours to complete. We
will post an additional notice when the upgrade has been completed.
[Jan 23, 2001, 3:19 PM] gort Upgrade
We are about to begin a drive swap on gort (www24). There will be
two brief periods of downtime, each approximately five minutes, at the
beginning and end of the maintenance. The server will remain online at all
other times. The entire upgrade could take up to 24 hours to complete. We
will post an additional notice when the upgrade has been completed.
[Jan 23, 2001, 2:52 PM] kirlian upgrade
We are about to begin a drive swap on kirlian (www110) in order to
improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 23, 2001, 7:53 AM] Billing and Upgrade Mail
Due to a misconfiguration, some customers sending mail to billing@pair.com
or upgrade@pair.com last night may have received bounce messages. If you
have received one, please resend your message. This error has been
corrected and will not recur.
[Jan 22, 2001, 12:10 AM] kodh maintenance
kodh (www90) was taken down for a chassis swap. Downtime was
aprox 10 minutes.
[Jan 21, 2001, 8:11 PM] zais Downtime
zais (www89) crashed due to high load. It required extensive filesystem
cleaning before being returned to normal service. Downtime was under 10
minutes.
[Jan 21, 2001, 6:28 PM] beeoro maintenance
After another unexplainable reboot, beeoro (www69) was taken down
briefly to swap possibly faulty memory. We shall continue to monitor the
situation, and swap to a new chasis if this server continues to show
problems.
[Jan 21, 2001, 4:13 PM] beeoro Downtime
beeoro (www69) crashed and was rebooted. A manual disk check was
performed. Total downtime was approxiamately 25 minutes.
[Jan 19, 2001, 8:20 PM] pi Upgrade Completed
We have completed the hard drive upgrade of pi. It now has a total of 30.7
GB of disk space. Downtime was under ten minutes.
[Jan 19, 2001, 1:15 PM] lepen reboot
lepen (www129) was rebooted to clear up a network problem, and has
returned to normal service again. Downtime was about 5 minutes.
[Jan 19, 2001, 1:10 PM] gamma maintenance completed
We have finished the hard drive swap on gamma (www2). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 19, 2001, 11:40 AM] scire maintenance completed
We have finished the hard drive swap on scire (www78). The
server now has 20 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 19, 2001, 2:15 AM] omicron Downtime
omicron (www9) crashed under high load and had to be rebooted.
Total downtime was under 5 minutes.
[Jan 19, 2001, 1:05 AM] gnaaste downtime
gnaaste (www79) crashed under high load, and had to be rebooted.
Total downtime was under 10 minutes.
[Jan 18, 2001, 10:23 PM] scire upgrade
We are about to begin a drive upgrade on scire (www78) to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when the
upgrade has been completed.
[Jan 18, 2001, 9:57 PM] muin upgrade complete
We have completed the hard drive upgrade of muin (www23). It now
has 30 GB of total hard drive space. Downtime was under ten minutes.
[Jan 18, 2001, 7:37 PM] pi upgrade
We have begun a drive upgrade on pi (www10) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when the
upgrade has been completed.
[Jan 18, 2001, 7:17 PM] thesel upgrade complete
We have completed the drive swap of thesel (www99). It now has a
total of 30 GB of space. Total downtime was under 10 minutes.
[Jan 18, 2001, 2:45 PM] muin maintenance
We are about to begin a drive upgrade on muin (www23) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 18, 2001, 2:32 PM] yekk maintenance completed
We have finished the hard drive swap on yekk (www124). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 18, 2001, 1:53 PM] gamma maintenace
We are about to begin a drive upgrade on gamma (www2) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 18, 2001, 11:36 AM] xrra maintenance completed
We have finished the hard drive swap on xrra (www116). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 18, 2001, 10:53 AM] yekk maintenance
We are about to begin a drive upgrade on yekk (www124) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 18, 2001, 10:33 AM] thesel maintenance
We are about to begin a drive upgrade on thesel (www99) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 18, 2001, 8:54 AM] kodh Maintenance
As part of preparations for today's FreeBSD -STABLE upgrade, the memory
on kodh (www90) has been swapped. Several recent crashes of the server
have been unexplained, and the problem was traced to faulty RAM.
The -STABLE upgrades will proceed shortly.
[Jan 18, 2001, 12:08 AM] UUnet Resolution
After almost precisely fourteen hours out of service, UUnet has restored
our OC-3c circuit to normal operation. We have switched traffic back from
the DS-3 circuit. We are holding our ticket with them open until we
receive a detailed explanation of what went wrong; early indications
suggest that a very minor error or accident occurred in their Pittsburgh
POP, and the fourteen hours was spent with UUnet, WorldCom, and Verizon
trying to decide who needed to do what to find the problem and fix it.
Needless to say, we are taking a long hard look at future plans for service
with UUnet. We have some excellent alternatives becoming available to us
for future growth paths. We will also be increasing circuit headroom
(there are two upgrades already long pending) so that any outage will lead
to less degradation of service.
[Jan 17, 2001, 6:12 PM] kodh Downtime
kodh (www90) crashed while in service requiring a manual disk check.
Downtime was 25 minutes.
[Jan 17, 2001, 6:09 PM] thuule maintenance completed
We have finished the hard drive swap on thuule (www49). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 17, 2001, 5:53 PM] fearn maintenance postponed
Due to problems with the new hard drive for fearn, the upgrade for
this server has been postponed.
There was a period of approximately ten minutes of downtime while the
server was reverted to its previous configuration.
[Jan 17, 2001, 2:33 PM] thuule maintenance
We are about to begin a drive upgrade on thuule (www49) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 17, 2001, 2:21 PM] ebad maintenance completed
We have finished the hard drive swap on ebad (www33). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 17, 2001, 1:56 PM] xrra maintenance
We are about to begin a drive upgrade on xrra (www116) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 17, 2001, 1:38 PM] coll maintenance completed
We have finished the hard drive swap on coll (www22). The
server now has 30 GB of hard disk space. Total downtime was under ten
minutes.
[Jan 17, 2001, 1:19 PM] UUnet Update
WorldCom has advised us that the problem with our circuit appears to be in
a segment provisioned by Verizon (formerly Bell Atlantic), and thus it is
being addressed through their support group. Although this means more
complications, it does indicate that there is progress in tracking down the
problem.
We are continuing to deliver our UUnet traffic through the shadow DS-3
circuit with some degradation of performance.
[Jan 17, 2001, 11:41 AM] UUnet Update
After 90 minutes of downtime, MCI WorldCom finally established the correct
department to handle our OC-3c outage. In the meantime, we have rerouted
all UUnet traffic to the spare DS-3; this will relieve overloading of other
circuits, although performance will still be slightly degraded.
We are continuing to work with WorldCom and UUnet engineers to get this
circuit restored to service. We are also continuing to push on the
deployment of our other pending circuits (specifically Sprint OC-3c and
AT&T DS-3).
[Jan 17, 2001, 11:22 AM] fearn maintenance
We are about to begin a drive upgrade on fearn (www40) in order
to improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 17, 2001, 11:03 AM] ebad maintenance
We are about to begin a drive upgrade on ebad (www33) in order to
improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 17, 2001, 10:43 AM] coll maintenance
We are about to begin a drive upgrade on coll (www22) in order to
improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 17, 2001, 10:19 AM] UUNet Outage
We are currently seeing a drastic reduction of the amount of traffic flowing on our UUNet circuit. All customer traffic is flowing over alternate paths at this time, and we are working with UUNet to find and fix the problem. Further details will be posted as they become available.
[Jan 17, 2001, 7:14 AM] bemnet downtime
bemnet(www158) crashed under high load and had to be restarted. Total
downtime was 5 minutes.
[Jan 17, 2001, 6:35 AM] kenaz downtime
kenaz (www138) crashed under high load and rebooted itself. Total downtime
5 minutes.
[Jan 17, 2001, 12:34 AM] bhoth downtime
A corrupt file caused an automatic reboot on bhoth (www163). After a
manual file system cleaning all is well. Total downtime 20 minutes.
[Jan 16, 2001, 9:38 PM] foma downtime
foma (www176) crashed under high load and rebooted. Total downtime
was under 5 mintues.
[Jan 16, 2001, 7:58 PM] enek upgrade complete
We have completed the drive swap of enek (www130). It now has 30 GB
of total disk space. Downtime was under 10 minutes.
[Jan 16, 2001, 6:31 PM] uruz Downtime
uruz (www135) crashed and required a manual reboot. Downtime was 10
minutes.
[Jan 16, 2001, 5:12 PM] dair maintenance completed
The hard drive upgrade of dair (www20) has been completed. The
server now has 30 GB of total disk space. At the same time, the server
was upgraded to an Athlon Thunderbird 1100 MHz, with 256 MB of RAM.
[Jan 16, 2001, 4:43 PM] upsilon maintenance completed
The hard drive upgrade on upsilon (www13) has been completed. The
server now has 30 GB of disk space. Downtime was under five minutes for
each period.
[Jan 16, 2001, 4:39 PM] beith Downtime
beith (www17) crashed due to a hardware failure. The system was brought back up after a reconfiguration and a thorough disk cleaning. Total downtime was approximately 30 minutes.
[Jan 16, 2001, 3:31 PM] enek maintenance
We are about to begin a drive upgrade on enek (www130) in order to
improve performance and storage. There will be two brief periods of
downtime, each approximately five minutes, at the beginning and end of the
maintenance. The server will remain online at all other times. The entire
upgrade could take up to 24 hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 16, 2001, 3:15 PM] tinne and psi maintenance completed
The hard drive upgrade of tinne (www21) has been completed. The
server now has 30 GB of total disk space. At the same time, the server was
upgraded to an Athlon Thunderbird 1100 MHz, with 256 MB of RAM.
Due to an administrative error, no notice was posted after the maintenace
on psi (www15) was completed. That server's maintenance has been
completed. The server now has 20 GB of total disk space. We apologize for
the inconvenience.
[Jan 16, 2001, 11:13 AM] tinne maintenance
We are about to begin a drive upgrade on tinne (www21) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 16, 2001, 10:50 AM] dair maintenance
We are about to begin a drive upgrade on dair (www20) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 16, 2001, 10:45 AM] oomur downtime
oomur (www209) crashed under heavy load, and was restarted. Scripts res
ponsible for the lock up have been locked, and the server has returned to normal
operation.
[Jan 16, 2001, 10:25 AM] upsilon maintenance
We are about to begin a drive upgrade on upsilon (www13) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 16, 2001, 10:00 AM] emra Downtime
emra (www183) crashed under load and was brought back online after
a filesystem check. Downtime was approximately 20 minutes.
[Jan 15, 2001, 10:37 PM] kappa upgrade complete
We have completed the system and drive upgrade of kappa (www6). The
server is now an Athlon Thunderbird 1100 MHz, with 256 MB of RAM and 30 GB of
space. Downtime was approximately ten minutes.
[Jan 15, 2001, 8:50 PM] huath upgrade complete
The drive upgrade of huath (www19) has been complete. The server
now has 30 GB of total space. Downtime was around 15 minutes.
[Jan 15, 2001, 3:37 PM] huath maintenance
We are about to begin a drive upgrade on huath (www19) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 15, 2001, 3:22 PM] dyyme maintenance completed
The hard drive upgrade on dyyme (www71) has been completed. The
server now has 20 GB of space total. Downtime was under five minutes.
[Jan 15, 2001, 11:45 AM] psi maintenance
We are about to begin a drive upgrade on psi (www15) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 15, 2001, 11:18 AM] kappa maintenance
We are about to begin a drive upgrade on kappa (www6) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 15, 2001, 10:51 AM] dyyme maintenance
We are about to begin a drive upgrade on dyyme (www71) in order to improve
performance and storage. There will be two brief periods of downtime, each
approximately five minutes, at the beginning and end of the maintenance.
The server will remain online at all other times. The entire upgrade could
take up to 24 hours to complete. We will post an additional notice when
the upgrade has been completed.
[Jan 14, 2001, 6:05 PM] lambe Downtime
lambe (www63) rebooted under heavy load, and has since returned to normal
operations. Downtime was less than 10 minutes.
[Jan 14, 2001, 4:22 PM] zais downtime
zais (www89) crashed under heavy load, and was manually rebooted. A manual
disk cleaning was performed before zais was brought back up. Total
downtime was approximately 20 minutes.
[Jan 12, 2001, 4:58 PM] kodh downtime
kodh (www90) rebooted under load, and has since returned to normal
operations. Downtime was less than 10 minutes.
[Jan 12, 2001, 4:26 AM] emra downtime
emra (www183) crashed under high load and had to be rebooted. Total
affected time, 10 minutes.
[Jan 11, 2001, 10:06 PM] omicron Downtime
omicron (www9) crashed due to high load. It has returned to
normal service, with a downtime of about 5 minutes.
[Jan 10, 2001, 12:41 PM] Brief Downtime
tinne (www21), coll (www22), and fearn (www40) will be down briefly for
an emergency physical relocation. Downtime for each server will be less
than five minutes. This relocation will also allow us to prepare for
upgrading these servers.
[Jan 9, 2001, 7:37 PM] beeoro Downtime
beeoro (www69) crashed under heavy load and had to be rebooted.
Total downtime was under 5 minutes.
[Jan 9, 2001, 5:24 PM] vert maintenance completed
We have completed the hardware swap on vert (www102). We fully
expect that this will eliminate any previous issues. Downtime was under
five minutes.
[Jan 9, 2001, 4:55 PM] further vert maintenance
Unfortunately, problems with vert (www102) have continued (not
related to RAM). We will be taking down this server shortly in order to
swap its general hardware.
[Jan 9, 2001, 1:08 PM] jera reboot
jera (www142) was rebooted to clear up a high load condition.
Downtime was approximately two minutes.
[Jan 9, 2001, 11:11 AM] vert maintenance completed
The RAM swap on vert has been completed. Downtime was
approximately two minutes.
[Jan 9, 2001, 10:54 AM] vert maintenance
Due to irregular behavior on vert (www102), we will be taking down
this server shortly to swap its RAM. We expect this to fully resolve any
lingering issues.
[Jan 9, 2001, 10:20 AM] zais Downtime
zais (www89) crashed under heavy load, and had to be manually rebooted and was given a manual disk cleaning before being brought back up. Total downtime was approximately 20 minutes.
[Jan 8, 2001, 4:34 PM] enda downtime
enda (www80) crashed under heavy load, and was brought back
online. Downtime was approximately 10 minutes.
[Jan 8, 2001, 9:56 AM] laguz downtime
laguz (www148) crashed under heavy load, and was returned to
service. Downtime was under 15 minutes.
[Jan 7, 2001, 5:03 PM] dhaame maintenance
After several unexplained server restarts, dhaame (www77) has been
swapped again to a new chasis in an attempt to isolate the source of the
problem. Downtime was less than 10 minutes, and this server is again
running smoothly.
[Jan 6, 2001, 6:41 AM] agho downtime
agho (www187) crashed under high load and was rebooted. Total downtime less
than 5 minutes.
[Jan 5, 2001, 6:03 PM] omicron maintenance completed
The upgrade of omicron has been completed. The server is now an
Athlon Thunderbird 1100 MHz, with 256 MB of RAM and 30 GB of space.
Downtime was approximately ten minutes.
[Jan 5, 2001, 1:17 PM] dhaame Downtime
dhaame (www177) crashed under heavy load, and was automatically rebooted. Total downtime was approximately 10 minutes.
[Jan 5, 2001, 5:26 AM] dhaame Restored
dhaame (www77) has been restored to service. The failure in the
new chassis was traced to a faulty power supply. After another chassis
swap, dhaame has the same new specifications (Athlon Thunderbird
1100 MHz), and is now running smoothly.
[Jan 5, 2001, 4:05 AM] dhaame Failure
dhaame (www77) has suffered a complete failure after the system
upgrade earlier this evening. We are swapping out the affected hardware,
and expect the server to be once again online within the hour.
[Jan 4, 2001, 11:24 PM] dhaame upgrade
dhaame (www77) was upgraded to an Athlon Thunderbird
1100 MHz, with 256 MB of RAM. Downtime was under five minutes.
[Jan 4, 2001, 11:03 PM] wredhor upgrade
wredhor (www76) was upgraded to an Athlon Thunderbird
1100 MHz, with 256 MB of RAM. Downtime was under five minutes.
[Jan 4, 2001, 10:45 PM] dhabat upgrade
dhabat (www105) was upgraded to an Athlon Thunderbird
1100 MHz, with 256 MB of RAM. Downtime was under five minutes.
[Jan 4, 2001, 10:21 PM] tenshian upgrade
tenshian (www103) was upgraded to an Athlon Thunderbird 1100 MHz, with
256 MB of RAM. Downtime was under five minutes.
[Jan 4, 2001, 1:03 AM] kodh downtime
kodh (www90) crashed under high load and had to be rebooted. Total downtime
approx. 10 minutes.
[Jan 3, 2001, 11:16 AM] vert logs truncated
Unfortunately, due to the RAM problems after the hardware upgrade of
vert (www102), the logs for January 2 are truncated. Logs will
appear normally for other dates. We apologize for the inconvenience.
[Jan 3, 2001, 11:05 AM] vert maintenance completed
The RAM swap on vert (www102) has been completed. We expect this
to resolve the matter. Downtime was under five minutes.
[Jan 3, 2001, 10:58 AM] vert maintenance
We have noticed minor problems ov vert (www102) following its
hardware upgrade. We will be taking down the server briefly to swap RAM.
Downtime will be minimal (under five minutes).
[Jan 2, 2001, 11:12 PM] omicron upgrade
We are about to begin a drive upgrade on omicron (www9) in order to improve
performance and storage. There will be two brief
periods of downtime, each
approximately five minutes, at the beginning and
end of the maintenance. The
server will remain online at all other times. The
entire upgrade could take up to 48
hours to complete. We will post an additional
notice when the upgrade has been completed.
[Jan 2, 2001, 11:08 PM] vert upgrade
vert (www102) was upgraded to an Athlon Thunderbird 1100 MHz, with
256 MB of RAM. Downtime was under five minutes.
[Jan 2, 2001, 10:51 PM] bellat upgrade
bellat (www101) was upgraded to an Athlon Thunderbird 1100 MHz,
with 256 MB of RAM. Downtime was under five minutes.
[Jan 2, 2001, 10:29 PM] zais upgrade
zais (www89) was upgraded to an Athlon Thunderbird 1100 MHz, with
256 MB of RAM. Downtime was under five minutes.
[Jan 2, 2001, 10:09 PM] beeoro upgrade
beeoro (www69) was upgraded to an Athlon
Thunderbird 1000 MHz, with 256 MB of RAM. Downtime was under five minutes.
[Jan 2, 2001, 10:09 AM] dair Downtime
dair (www20) crashed under heavy load, and automatically rebooted. Total downtime was approximately 10 minutes.
View Notices for 2000
