[Dec 31, 2002, 11:15 PM] muin Planned Maintenance

muin (www23) will be taken down after midnight for a chassis swap. Downtime should be under 10 minutes. An additional notices with the new hardware details will be posted when complete.

[Dec 31, 2002, 9:19 AM] muin Downtime

muin (www23) was rebooted to clear up an internal problem with system logging. Total downtime was less than five minutes.

[Dec 31, 2002, 12:50 AM] bhoth Maintenance Completed

The bhoth upgrades have been completed. It has been upgraded to an Athlon XP 2100, with 512MB of RAM, and an 80GB disk. The shadow drive was also installed at this time. Total downtime was about 20 minutes.

[Dec 30, 2002, 10:56 PM] bhoth Planned Maintenance

bhoth (www163) will be taken down shortly to begin a drive upgrade. There should be a downtime to start the swap of about 5 minutes. Also, when the drive upgrade is completed, there will be a second downtime of about 15 minutes. This will be to complete the swap, as well as upgrade the chassis. An additional notice will be posted with the new hardware information when complete.

[Dec 30, 2002, 11:48 AM] dsumaui Downtime

dsumaui (www275) crashed under high load and had to be rebooted. Total downtime was under 10 minutes.

[Dec 29, 2002, 12:03 PM] bhoth Downtime

bhoth (www163) crashed after running out of swap space, and required a reboot. During the file system check, we discovered a hard drive error. A drive swap has been scheduled. We will post more information as it becomes available.

[Dec 29, 2002, 7:02 AM] quesse Planned Maintenance

We will be taking quesse (www44) down briefly this morning to replace it's shadow drive. Total downtime for this maintenance should be under 10 minutes.

[Dec 29, 2002, 12:34 AM] hyarmen Downtime

After a crash and reboot, system errors were found and hyarmen (www66) was given a chassis swap. The server is now an Athlon XP2100+ with 512MB RAM.

[Dec 24, 2002, 4:28 AM] ungwe Maintenance Completed

The drive swap of ungwe has been completed at this time. Additionally, this server has been upgraded to an Athlon 2100 system.

[Dec 24, 2002, 12:40 AM] ungwe Emergency Maintenance

ungwe (www48) crashed after running out of swap space, and required a reboot. During the file system check, we discovered hard drive errors, and have began an emergency drive swap. An additional notice will be posted when the swap is complete.

[Dec 23, 2002, 1:17 PM] Webmail Outgoing Mail

We have found and fixed a problem with outgoing mail on one of the three servers that power our webmail.pair.com service. The delivery of around 600 messages had been delayed -- we expect these messages to be sent within the next 15 minutes. This problem should not recur. We apologize for the inconvenience.

[Dec 20, 2002, 4:50 PM] yekk Downtime

yekk (www124) crashed under high load and had to be rebooted. Total downtime about 15 minutes.

[Dec 20, 2002, 6:53 AM] Network Maintenance

Genuity has advised us that they are currently doing maintenance on our circuit. All traffic is being carried as normal through alternate providers.

[Dec 19, 2002, 8:32 PM] sowilu, enek Maintenance Completed

We have completed the chassis upgrades of sowilu (www145) and enek (www130). Both servers are now Athlon XP 2100MHz systems with 512MB of RAM. Total downtime per server was under 10 minutes.

[Dec 19, 2002, 4:48 PM] Planned Maintenance

Tonight after 6PM EST the following servers will receive chassis upgrades:

enek (www130)
sowilu (www145)

Total downtime for this maintenance should be under 10 minutes for each server. Additional notices will be posted upon the completion of the maintenance.

[Dec 19, 2002, 4:46 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

flure (www75)
wutare (www277)
tyllur (www285)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 19, 2002, 11:13 AM] ACC Maintenance Complete

The maintenance has been completed, and the Account Control Center has been reopened to customer access.

[Dec 19, 2002, 9:05 AM] Account Control Center Maintenance

The Account Control Center maintenance announced yesterday has now begun. We will post another notice when it has been completed.

[Dec 19, 2002, 3:41 AM] Relay Upgrade Completed

We have completed the hardware and software upgrades scheduled for our relay service. The service is running more smoothly than ever. Queued e-mail will continue to be delivered from the old server for a few days. We are also working on plans to further expand the relay service for increased performance and reliability.

[Dec 18, 2002, 3:13 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

hyarmen (www66)
dhaame (www77)
muyal (www236)
iruvul (www239)
xamsa (www244)
seyyav (www238)
zortzi (www243)
vondu (www241)
aqquz (www248)
wokkil (www249)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 18, 2002, 11:19 AM] ACC Maintenance -- correction

The Account Control Center maintenance announced in the last notice will take place on December 19, 2002, not the 18th as stated in the notice. We apologize for the error.

[Dec 18, 2002, 11:04 AM] Account Control Center Maintenance

The Account Control Center will be taken offline tomorrow, December 18, 2002, for maintenance to our back-end systems. The maintenance will begin at approximately 6:00 AM EST, and should be completed within 4 hours. Additional notices will be posted when the maintenance period starts, and when it is completed.

[Dec 17, 2002, 6:36 PM] relay Service

Today we again experienced a denial-of-service attack on our outgoing SMTP relay service, due primarily to a very large, ill-advised mailing by a customer.

We have been testing a replacement for the relay service, utilizing upgraded hardware and software. This will be swapped into service one evening this week, with no disruption expected. We will post a notice when this upgrade is complete. We are also making plans in the near term to expand this service to multiple servers sharing the load, at which point service degradations should no longer occur.

Please accept our apologies for the recent difficulties with this service.

[Dec 17, 2002, 5:16 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

sibum (www224)
eahta (www227)
alpha (www1)
tehun (www226)
petora (www229)
uinus (www228)
dieci (www231)
patru (www232)
peswar (www233)
okkod (www235)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 16, 2002, 3:16 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

gnaaste (www79)
aster (www96)
scire (www78)
fimfi (www223)
coll (www22)
ydhu (www83)
zirx (www106)
vilya (www60)
thesel (www99)
niwun (www225)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 14, 2002, 3:03 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

nuumen (www55)
xrra (www116)
penkwe (www220)
xebte (www230)
aedde (www84)
paat (www100)
yuzz (www117)
falku (www98)
ainaz (www221)
fithwor (www222)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 14, 2002, 7:40 AM] delsi Emergency Maintenance

delsi (www206) was taken down to swap the RAM. This will hopefully correct the instability it has been showing lately. Total downtime was about 5 minutes.

[Dec 13, 2002, 6:59 PM] zhiri Downtime

Zhiri (www290) was rebooted this evening to resolve a problem with its network card that was causing intermittent outages. Downtime was less than 5 minutes.

[Dec 13, 2002, 1:54 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

naih (www214)
tenshian (www103)
zais (www89)
emlis (www216)
dair (www20)
tinne (www21)
kappa (www6)
upsilon (www13)
wawrra (www109)
duwo (www219)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 13, 2002, 6:36 AM] thurisaz Maintenance Completed

The drive swap for thurisaz has been completed. It now has an 80GB drive. Also, we took this opportunity to upgrade the chassis. It is now an Athlon XP 1600MHz with 512MB of RAM. Total downtime for this was about 20 minutes.

[Dec 13, 2002, 4:44 AM] fearn Downtime

In order to stabilize fearn (www40), we are performing a chassis swap.

[Dec 13, 2002, 4:29 AM] fearn Downtime

fearn (www40) crashed and required a filesystem cleaning. It has been returned to stable service.

[Dec 13, 2002, 3:44 AM] thurisaz Emergency Maintenance

We have detected errors on the hard drive of thurisaz (www136) and will be taking it down to start a drive swap. There will be 2 breif periods of dowtime, at the start and completion of the swap. An additional notice will be posted when the swap is complete.

[Dec 12, 2002, 3:05 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

imtis (www208)
oomur (www209)
pyyl (www97)
enda (www80)
amse (www210)
omega (www16)
flaid (www91)
erraj (www213)
kaazh (www212)
akket (www211)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 12, 2002, 3:18 AM] fearn Downtime

fearn (www40) became unresponsive and required a reboot. Total downtime was about 10 minutes.

[Dec 12, 2002, 1:48 AM] athani Emergency Maintenance

After 2 repeated crashes, athani has received a new chassis. It is now an Athlon XP2100 with 512MB of RAM. Downtime was about 20 minutes.

[Dec 11, 2002, 3:08 PM] Problems with Relay Service

Our relay service has been swamped by an ill-advised newsletter mailing by a customer this morning. We have been working to clear the queue and ensure prompt delivery of all messages sent through relay. The queue is steadily improving at this time and the situation is under control.

We have a hardware and software upgrade scheduled for relay early next week. Further details will be posted when the upgrade occurs; no downtime is expected.

[Dec 11, 2002, 2:41 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

vala (www59)
gwind (www74)
noldo (www56)
shen (www88)
epsilon (www3)
maborym (www204)
mildh (www87)
straif (www26)
delsi (www206)
rho (www11)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 11, 2002, 11:35 AM] fearn Downtime

fearn(www40) crashed and required a file system check before coming back online. Downtime was under 10 minutes.

[Dec 10, 2002, 6:13 PM] athani Downtime

athani (www157) crashed, and required an extensive filesystem check before it was returned to service. Downtime was less than 15 minutes.

[Dec 10, 2002, 4:20 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

namt (www196)
quesse (www44)
thuule (www49)
allatu (www197)
klesh (www200)
morx (www198)
maym (www202)
humph (wwww118)
tama (www70)
dyyme (www71)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 10, 2002, 10:02 AM] athani Downtime

athani(www157) crashed and required an extensive file system check before coming back online. Downtime was approximately 25 minutes.

[Dec 9, 2002, 7:59 PM] ceirt Maintenance Completed

We have completed the drive swap on ceirt (www39). Total downtime for this maintenance was under 20 minutes.

[Dec 9, 2002, 3:23 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

arda (www62)
silme (www64)
ilwe (www81)
aaze (www65)
cuzea (www94)
isris (www194)
nwalme (www57)
unque (www54)
roomen (www61)
ungwe (www48)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 9, 2002, 2:31 PM] namt Downtime

namt(www196) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Dec 8, 2002, 12:43 PM] reit Emergency Maintenance

This morning we discovered that the fan in reit (www108) was dying. We took the server down on an emergency basis for a full chassis swap. It is now a Pentium 4 1.8ghz with 256MB. Total downtime for this maintenance was under 15 minutes.

[Dec 7, 2002, 11:48 PM] kodh Planned Maintenance

kodh (www90) will be taken down to install an additional 512MB of RAM, bringing the total to 1024MB. Downtime should be under 5 minutes.

[Dec 7, 2002, 2:24 PM] kodh Emergency Maintenance

kodh (www90) experienced a warning condition which required an emegency chassis swap. There was a 10 minute downtime while the swap occured. kodh has been upgraded to a Pentium 4 1.7 Ghz with 512 MB of RAM.

[Dec 7, 2002, 12:40 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

ifin (www36)
csor (www190)
edad (www31)
xi (www8)
ilceille (www189)
aveu (www192)
virre (www193)
parma (www42)
idad (www32)
poict (www195)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 7, 2002, 1:25 AM] ceirt Emergency Maintenance

We have observed a warning condition on the hard drive in ceirt (www39) which may indicate an imminent failure. We will be taking this drive down immediately to begin a hard drive swap. There will be two brief periods of downtime at the beginning and end of this maintenance. We will post additional information as it becomes available.

[Dec 6, 2002, 4:53 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

fiann (www188)
muin (www23)
omicron (www9)
sothor (www182)
aerre (www185)
linato (www186)
emra (www183)
ur (www30)
khin (www184)
ebad (www33)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 6, 2002, 9:53 AM] Server Maintenance

ncftpd (the FTP daemon) will be restarted on all user servers this morning. This will result in a brief interruption in FTP availability, which should be less than 2 minutes per server.

[Dec 5, 2002, 10:49 PM] sallane Maintenance Completed

The drive swap to sallane (www263) has been completed. The hard drive has been upgraded to 80 GB.

[Dec 5, 2002, 9:51 PM] glikk Maintenance Completed

The drive swap of glikk has been completed at this time. Total downtime was less than 10 minutes.

[Dec 5, 2002, 8:46 PM] sallane Emergency Maintenance

sallane (www263) started showing signs of a drive failure. An emergency drive swap was started. Also during this time, sallane received a chassis swap. It has been upgraded to a Celeron 1.7 Ghz with 512 MB of RAM. Another notice will be posted once the drive swap is completed.

[Dec 5, 2002, 2:15 PM] RAM Upgrades and Maintenance

Tonight after 6PM EST the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

foma (www176)
vsael (www173)
anga (www47)
zulle (www175)
biont (www174)
gief (www178)
londaste (www177)
raitax (www171)
ipre (www181)
merull (www180)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 5, 2002, 11:57 AM] glikk Emergency Maintenance

Due to the presence of failing sectors, a drive swap has been initiated on glikk to replace the failing drive. There will be another short downtime once this work has been completed. Due to the location of the failing sectors, we do not expect any loss of user data during this operation.

[Dec 5, 2002, 1:39 AM] cicka, qenni, xomi Maintenance Completed

cicka, qenni, and xomi have all received chassis swaps during their planned upgrades. cicka and xomi are now P3 1000MHz, with 512MB of RAM. qenni is now an Athlon 850, with 512MB of RAM. Downtime for the chassis, RAM, and drive upgrades was about 15 minutes for each server.

[Dec 4, 2002, 8:58 PM] whury Maintenance Completed

During the course of the RAM upgrade, whury (www179) suffered a dying CPU fan. The server was given a full chassis swap, and is now a Pentium 1 GHz with 512MB of RAM. Approximate downtime was 20 minutes.

[Dec 4, 2002, 7:42 PM] RAM Upgrades and Maintenance

Tonight, the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

neter (www132)
kanat (www128)
whury (www179)
fearn (www40)
ailm (www28)
ceirt (www39)
bemnet (www158)
qenni (www168)
xomi (www169)
cicka (www167)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 3, 2002, 10:54 PM] lissa, neled, tiwaz Planned Maintenance

During the course of the RAM upgrades, the following servers were given full chassis swaps:

tiwaz (www146) - Pentium 4 2000MHz, 512MB RAM
lissa (www72) - Pentium 4 2000MHz, 512MB RAM
neled (www127) - Pentium 4 2000MHz, 512MB RAM

[Dec 3, 2002, 10:48 PM] mini Emergency Maintenance

mini (www126) experienced a failed fan and required an emergency chassis swap. It is now a Pentium 4 2000 with 512MB if RAM. Total downtime was under 10 minutes.

[Dec 3, 2002, 8:36 PM] RAM Upgrades and Maintenance

Tonight, the following servers will be upgraded to 512MB of RAM, and receive a new Shadow Drive:

flunn (www123)
yekk (www124)
ruis (www27)
athani (www157)
thnad (www121)
sowilu (www145)
tiwaz (www146)
lissa (www72)
enek (www130)
neled (www127)

These upgrades were announced in our November newsletter. There will be a brief downtime of about 10 minutes for each server while the new hardware is added.

[Dec 2, 2002, 2:57 PM] AT&T Network Maintenance

AT&T has advised us that they will be performing emergency maintenance on their Philadelphia POP, where our OC-3c circuit terminates, this evening between midnight and 6am Eastern time. They are again replacing equipment which they suspect to be faulty.

During this maintenance, customer traffic will be carried by our other providers with no impact on performance expected.

[Dec 2, 2002, 12:17 PM] dair Downtime

dair(www20) became unresponsive and was rebooted. Downtime was around 10 minutes.

[Nov 30, 2002, 5:48 PM] sallane Downtime

sallane (www263) was down for approximately 15 minutes this evening. Upon investigation we discovered a bad RAM chip. It has been replaced, and the server has returned to normal operation.

[Nov 30, 2002, 12:26 AM] kodh Downtime

kodh (www90) crashed and had to be rebooted. Total downtime was about 10 minutes.

[Nov 29, 2002, 3:17 PM] iski Downtime

iski (www242) encountered a resource problem and had to be rebooted. Total downtime was approximately five minutes.

[Nov 29, 2002, 1:57 PM] diviy Emergency Maintenance

diviy was briefly brought down to perform a file systems check in response to some file corruptoin that was discovered. The check passed without any problems. Total downtime was approximately 5 minutes.

[Nov 25, 2002, 4:06 AM] penkwe Downtime

penkwe (www220) crashed and was rebooted. Total downtime was about 10 minutes.

[Nov 23, 2002, 5:48 PM] sallane Downtime

sallane went down this evening at around 17:24:52 and was back up at around 17:45:30. The downtime was the result of the drive cable being unseated. Reseating the drive cable corrected the problem.

[Nov 22, 2002, 2:08 AM] kappa Downtime

kappa(www6) crashed and was rebooted. Total downtime was about 10 minutes.

[Nov 21, 2002, 8:45 PM] kappa Downtime

kappa (www4) crashed under high load and had to be rebooted. After a manual file system cleaning it came back up with no problems.

[Nov 21, 2002, 10:23 AM] yanta Emergency Maintenance

This morning we discovered the fan in yanta (www67) was dying. We took the server down on an emergency basis for a full chassis swap. It is now a Pentium 4 1.7GHz with 512Mb of RAM. Total downtime for this maintenance was about 10 minutes.

[Nov 20, 2002, 12:35 PM] AT&T Network Maintenance

AT&T has advised us that they have accelerated planned network maintenance for the Philadelphia POP where our circuit terminates, in light of the recent outages at that location. They will be using some portion of their maintenance window, which runs from midnight tonight to 6am, Eastern time, to replace faulty equipment. We expect a relatively short downtime on the AT&T circuit, which will not affect the overall performance of our network.

We hope that this will resolve the problems AT&T has recently had.

[Nov 19, 2002, 5:20 PM] wredhor Downtime

After several unexpected reboots, the hardware for wredhor has been swapped to a new Pentium 4 chasis. We will continue to monitor this server for problems, should this upgrade not resolve the issue.

[Nov 19, 2002, 4:34 PM] AT&T Network Problem

Our AT&T circuit experienced a brief fourteen minute downtime, after AT&T had a network impairment with their Philadelphia router. Please accept our apologies for any inconvenience this problem may have caused.

[Nov 19, 2002, 12:01 PM] ailm Downtime

ailm(www28) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 19, 2002, 11:32 AM] omicron Downtime

omicron(www9) crashed and was rebooted. Downtime was less than 10 minutes.

[Nov 19, 2002, 9:17 AM] ns000.ns0.com Maintenance Completed

The upgrade of ns000.ns0.com has been completed. Downtime was approximately 15 minutes.

[Nov 19, 2002, 8:56 AM] ns000.ns0.com Emergency Maintenance

ns000.ns0.com will be taken down this morning for a hardware upgrade. Downtime is expected to be less than 20 minutes.

[Nov 18, 2002, 6:49 PM] emancholl Downtime

emancholl (www37) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 18, 2002, 8:18 AM] midhae Downtime

midhae crashed and required an extensive file system check before coming back online. Total downtime was around 20 minutes.

[Nov 18, 2002, 5:56 AM] Network Problems

At 5:20 AM Eastern time, we experienced a short internal network problem. This problem affected about 30 of our servers, and lasted approximately 10 minutes. The problem is currently being investigated, and we will post more information once we know more about the problem.

[Nov 16, 2002, 5:46 PM] sallane Emergency Maintenance

Sallane went down at 17:23:51 and subsequently received a memory swap. It went back up at around 17:45:30 for a total downtime of around 23 minutes.

[Nov 16, 2002, 2:19 PM] midhae Downtime

midhae (www278) was rebooted to clear up a problem with the network interface. Downtime was under 15 minutes.

[Nov 16, 2002, 1:54 PM] ns000 Underwent Emergency Maintenance

Due to scripting issues, ns000 was unavailable for a period of time overnight. This has now been resolved, and it is now back in service. Please accept our apologies for the inconvenience.

[Nov 14, 2002, 12:10 PM] Secondary Nameservice Problems Resolved

One of our secondary nameservers, ns000.ns0.com, experienced problems overnight with domain resolution. As a result, some domains using that nameserver may have been intermittently unavailable to some viewers, although the effect was lessened by working primary nameservice as well as DNS caching at external locations.

We have located and fixed the cause of the problem, and it should not recur. We apologize for the inconvenience.

[Nov 13, 2002, 9:31 AM] ailm Downtime

ailm(www28) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 13, 2002, 3:48 AM] qaske Maintenance Completed

The chassis and drive upgrades of qaske have been completed. It is now a Pentium4 2Ghz, with 512MB of RAM, and a 60GB drive. Total downtime was about 15 minutes.

[Nov 13, 2002, 12:11 AM] qaske Planned Maintenance

qaske (www250) will be taken down shortly to receive a chassis, and drive upgrade. There will be 2 short periods of downtime, to start, and complete the upgrade. Each should be no more then 10 minutes. An additional notice will be posted when the upgrade is completed.

[Nov 12, 2002, 2:53 PM] SSH Updated

We have deployed the latest version of OpenSSH, 3.5p1, to our servers this afternoon. This routine maintenance ensures that we are up-to-date with any related security vulnerabilities. Active connections through SSH were disrupted by this upgrade; we apologize for the inconvenience. Please note that no server keys have been changed, however.

[Nov 12, 2002, 2:15 AM] relay downtime

For about 40 minutes, our relay server, relay.pair.com, was unable to accept connections. After extensive troubleshooting and a file system check, this server is now back in service. We will continue to monitor this server closely.

[Nov 11, 2002, 5:09 PM] Network Problems Resolved

The problems with inbound traffic from Road Runner over our AT&T circuit are now resolved. We will continue to monitor for any further problems.

[Nov 11, 2002, 2:38 PM] Network Problems

We are experiencing problems with Internet traffic over our AT&T circuit. We have contacted AT&T regarding the problem and they are investigating the situation. The problem appears to be isolated to inbound traffic from Road Runner. We appreciate your patience and will post additional information when it becomes available.

[Nov 11, 2002, 2:01 PM] mu Maintenance Completed

The chassis swap of mu (www7) has been completed. Downtime was approximately five minutes. The server is now a Pentium 4 2000MHz, with 512MB RAM.

[Nov 11, 2002, 1:39 PM] mu Emergency Maintenance

We will be taking down mu (www7) on an emergency basis for a chassis swap. We believe that a hardware failure is to blame for the recent crashed. Downtime should be no more than 15 minutes. More information will be posted when it becomes available.

[Nov 11, 2002, 12:38 PM] mu Downtime

mu (www7) crashed and required a file systems check before coming back on line. Downtime was under 10 minutes.

[Nov 10, 2002, 10:38 PM] mu Downtime

mu (www7) was down this evening for approximately 15 minutes to clear out some system problems. It has since returned to normal operation. However, due to the problems last week with this server, we will be taking another look at it to attempt to find any lingering issues that might be causing problems.

[Nov 9, 2002, 2:25 PM] klesh Downtime

Klesh came under heavy load and needed to be rebooted. Downtime was approximately 10 minutes or less.

[Nov 8, 2002, 9:46 PM] ailm Downtime

ailm (www28) crashed under high load and had to be rebooted. Total downtime was less than 5 minutes.

[Nov 8, 2002, 9:34 PM] mu Maintenance Completed

mu's driveswap was successfully completed.

[Nov 8, 2002, 7:36 PM] mu Downtime & Emergency Maintenance

mu (www7) crashed under high load for the second time in as many days, and had to have it's file system extensively cleaned again. We are initiating an emergency driveswap, during which it will be taken down briefly for an upgrade twice. Total downtime should be around 20 minutes.

[Nov 8, 2002, 5:55 PM] nish Planned Maintenance

nish will be taken down briefly for an upgrade to system memory to 512MB. Downtime should be less than 10 minutes.

[Nov 8, 2002, 11:52 AM] kanat Downtime

kanat (www128) crashed under a high load and was rebooted. The downtime was under 15 minutes.

[Nov 7, 2002, 10:42 PM] halla Planned Maintenance

In order to increase preformance and stability, halla has begun a drive and chasis upgrade to a P4 2Ghz system with 60GB of drive space. There will be another brief downtime once the drive swap has been completed later this evening.

[Nov 7, 2002, 4:06 PM] mu Downtime

mu crashed under load, and required an extensive filesystem cleanup before it was returned to service. Downtime was approximately 15 minutes.

[Nov 7, 2002, 2:35 PM] halla Downtime

halla became unresponsive and was rebooted. Downtime was less than 10 minutes.

[Nov 7, 2002, 3:50 AM] gamma Maintenance Completed

The chassis and drive upgrade for gamma has been completed. It is now a Pentium4 2000MHz, with 512MB of RAM, and a 60GB hard drive. Total downtime was about 15 minutes.

[Nov 6, 2002, 11:03 PM] gamma Planned Maintenance

gamma (www2) will be taken down shortly for a drive, and chassis upgrade. There will be to short periods of downtime, to start and complete the upgrade. Downtime for each should be less then 10 minutes. An additional notice will be posted when the upgrade is complete.

[Nov 6, 2002, 10:53 PM] lambe Emergency Maintenance

lambe (www63) has received an emergency chassis swap after we noticed seemingly random reboots. It is now a Pentium4 2000MHz, with 512MB of RAM. Total downtime was about 10 minutes.

[Nov 6, 2002, 4:00 AM] mu Maintenance Completed

The drive upgrade of mu has been completed. It now has a 60GB drive. Total downtime was about 10 minutes.

[Nov 6, 2002, 2:34 AM] zatz Maintenance Completed

The drive swap of zatz has been completed. It now has a 60GB drive. Total downtime was about 10 minutes.

[Nov 6, 2002, 12:14 AM] mu, zatz Planned Maintenance

mu (www7) and zatz (www122) will be taken down shortly to begin drive upgrades. There will be to breif periods of downtime, less then 10 minutes each. An additional notice will be posted when the swaps have been completed.

[Nov 5, 2002, 8:55 PM] halla Downtime

halla(www68) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 5, 2002, 3:31 PM] halla Downtime

halla(www68) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 4, 2002, 3:15 PM] neter Downtime

neter crashed and was rebooted. Downtime was under 10 minutes.

[Nov 4, 2002, 2:29 PM] halla Downtime

halla(www68) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 4, 2002, 12:44 PM] dyyme Downtime

dyyme (www71) became unresponsive and was rebooted. Downtime was less than 10 minutes.

[Nov 4, 2002, 3:23 AM] mu Downtime

mu (www7) crashed under high load and was rebooted. Total downtime was approximately 15 minutes.

[Nov 3, 2002, 4:31 AM] zulle Downtime

zulle (www175) became unresponsive, and required a reboot. Downtime was about 15 minutes.

[Nov 1, 2002, 10:27 AM] relay.pair.com

Relay.pair.com crashed earlier this morning and was rebooted. Downtime was about 30 minutes. We apologize for any inconvenience this situation may have caused our customers.

[Nov 1, 2002, 9:46 AM] huath Downtime

huath became unresponsive and was rebooted. Downtime was under 10 minutes.

[Nov 1, 2002, 5:28 AM] emancholl Maintenance Completed

The drive swap for emancholl has been completed. Total downtime for this was about 10 minutes.

[Nov 1, 2002, 1:12 AM] torbaane Maintenance Completed

The drive upgrade for torbaane has been completed. Total downtime for the drive swap, and chassis upgrade was about 15 minutes.

[Nov 1, 2002, 12:43 AM] emancholl Planned Maintenance

In order to improve system performance, a drive swap has been started on emancholl to a faster drive model, as well as to increase available storage. There will be another brief downtime once this has been completed.

[Nov 1, 2002, 12:39 AM] gamma, mu, zatz Planned Maintenance

As a part of ongoing upgrades the following servers were upgraded tonight to improve system performance:

• gamma - Pentium III 1000 512MB RAM
• mu - Athlon 850 512MB RAM
• zatz - Athlon 850 512MB RAM

Downtime in each case was less than 10 minutes.

[Oct 31, 2002, 10:57 PM] torbaane Planned Maintenance

We have begun a drive swap and upgrade for torbaane this evening to improve performance and as a protective measure, as we have seen signs that the drive on torbaane may be beginning to fail. There will be one more brief outage when the drive swap has been completed. The new chassis for torbaane is an Athlon 1400Mhz with a 60GB drive.

[Oct 31, 2002, 2:35 PM] xebte Downtime

xebte(www230) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Oct 29, 2002, 6:51 PM] ailm Downtime

ailm (www28) was rebooted to clear up a problem with the network interface. Downtime was less than 10 minutes.

[Oct 28, 2002, 9:56 PM] halla Downtime

halla (www68) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Oct 28, 2002, 1:28 PM] yitiya Downtime

yitiya (www252) was rebooted to clear up a resource shortage. Downtime was approximately one minute.

[Oct 27, 2002, 7:08 AM] tama Downtime

tama (www70) became unrepsonsive and was rebooted. Downtime was less than 10 minutes.

[Oct 25, 2002, 12:03 AM] quan, vepar Planned Maintenance

quan and vepar were taken down briefly this evening for an upgrade of system hardware to Pentium 4 1.8Ghz systems. Downtime for each was approximately 5 minutes.

[Oct 23, 2002, 10:00 AM] terfideu Downtime

terfideu became unrepsonsive and was rebooted. Downtime was less than 10 minutes.

[Oct 22, 2002, 9:55 PM] phpWebSite Security Vulnerability

A vulnerability in the phpWebSite software package has been announced at http://www.pair.com/pair/support/notices/security.html If you use this software, please upgrade to version 0.8.3 or later, available from http://phpwebsite.appstate.edu/

[Oct 21, 2002, 11:12 PM] torbaane Downtime

torbaane (www262) crashed under high load and had to be rebooted. Total downtime was approximately 25 minutes.

[Oct 21, 2002, 8:04 PM] halla Downtime

halla (www68) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Oct 21, 2002, 4:14 PM] Network Problems Resolved

Between 3:46pm and 3:59pm Eastern time, some customers may have experienced degradation of network performance. We tracked this to a routing anomaly related to the Pittsburgh Internet Exchange. We are now taking steps to prevent any recurrence of this anomaly. No further service impact is expected.

[Oct 21, 2002, 12:02 PM] news.pair.com Maintenance Completed

news.pair.com has been restored to service at this time.

[Oct 21, 2002, 11:22 AM] news.pair.com Downtime

The pair NNTP news server, news.pair.com, has developed problems and is currently unavailable. We are in the process of restoring it to service. We will post an additional notice when more information is available.

[Oct 21, 2002, 10:40 AM] db11 Downtime

db11 froze and was rebooted. Total downtime was approximately 10 minutes.

[Oct 18, 2002, 12:18 PM] DB Server 4.6 Upgrades Completed

Today's upgrades have been completed. All shared db servers are now running FreeBSD 4.6-STABLE.

[Oct 18, 2002, 10:19 AM] DB Server 4.6 Planned Upgrades

The following MySQL servers will be upgraded to FreeBSD 4.6 today: db db2 db3 db4 db5 db6 db7 db8 db9 db10 db11 db12 db14 db15 db16 db17 Downtime should be less than five minutes per server.

[Oct 18, 2002, 4:50 AM] delsi Maintenance Completed

The drive upgrade of delsi has been completed. It now has a 60GB drive. Total downtime was about 10 minutes.

[Oct 18, 2002, 2:53 AM] morx Maintenance Completed

The drive upgrade of morx has been completed. It is now a 60GB drive. Total downtime was about 10 minutes.

[Oct 17, 2002, 10:50 PM] morx, delsi Planned Maintenance

morx (www198) and delsi (www206) will be taken down shortly to begin drive upgrades. There should be 2 short periods of downtime, and the beginning and completion of the upgrade. The downtimes should be no more then 10 minutes each. An additional notice will be posted when the upgrades are complete.

[Oct 17, 2002, 4:58 PM] DB Server 4.6 Upgrades

The following database servers have all been upgraded to FreeBSD 4.6:

db13
db18
db19
db20
db21
db22
db23
db24
db25
db26
db27
db28
db29
db30
db31
db32
db33
db34
db35
db36
db37
db38

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately.

[Oct 17, 2002, 4:00 AM] kayan Maintenance Completed

The drive upgrade of kayan has been completed. The chassis was also updated at this time to a Pentium4, 1.8GHz, with 512MB of RAM. Total downtime was about 20 minutes.

[Oct 17, 2002, 2:00 AM] fehu Maintenance Completed

The drive upgrade for fehu has been completed. It now has a 60GB drive. Total downtime was about 10 minutes.

[Oct 16, 2002, 11:14 PM] kayan, fehu Planned Maintenance

kayan (www133) and fehu (www134) will be taken down shortly to begin drive upgrades. There will be 2 short periods of downtime, at the start and completion of the swaps. Each should be under 10 minutes. An additional notice will be posted when the swaps have been completed.

[Oct 16, 2002, 2:12 PM] torbaane Downtime

torbaane(www262) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Oct 16, 2002, 12:10 PM] Signup System Maintenance

Our signup system was taken offline for approximately 10 minutes this morning for maintenance. It is now operating normally.

[Oct 16, 2002, 3:37 AM] ipre Maintenance Completed

The drive upgrade for ipre has been completed. It now has a 60GB drive. Total dowtime was about 15 minutes.

[Oct 16, 2002, 3:10 AM] flunn Maintenance Completed

The drive swap on flunn has been completed. It now has a 60GB drive. We also took this opportunity to upgrade the chassis. It is now a Pentium4 1.8GHz, with 512MB of RAM. Total downtime was about 20 minutes.

[Oct 15, 2002, 11:29 PM] flunn, ipre Planned Maintenance

flunn (www123) and ipre (www181) will be taken down shortly to begin drive upgrades. There will be 2 breif periods of downtime, at the start and completion of the swaps. Each period of downtime should be no longer then 10 minutes. An additional notice will be posted when the swaps are completed.

[Oct 15, 2002, 1:45 PM] quan Downtime

quan(www120) crashed due to high load and was rebooted. Downtime was under 10 minutes.

[Oct 15, 2002, 1:23 PM] cele Maintenance Completed

The emergency hard drive swap for cele (www73) has been completed. Downtime was approximately 20 minutes. After the server was brought up with a new drive, content was linked to the old drive as it was copied over, in order to minimize downtime.

[Oct 15, 2002, 11:05 AM] cele Emergency Maintenance

Cele crashed and could not be brought back up. A drive swap is being performed on it now. Downtime during the drive swap in unknown at this time. Further details will be posted when available.

[Oct 14, 2002, 8:51 AM] nish Downtime

nish(www92) became unresponsive and was rebooted. Total downtime was less than 10 minutes.

[Oct 13, 2002, 4:09 PM] wutare Downtime

wutare (www277) crashed under high load and was rebooted. Downtime was under 10 minutes.

[Oct 11, 2002, 5:04 PM] tuzuba Downtime

tuzuba (www256) crashed under high load and had to be rebooted. Total downtime was less than 5 minutes.

[Oct 11, 2002, 11:05 AM] nish Downtime

nish(www92) crashed and was rebooted. Downtime was under 10 minutes.

[Oct 11, 2002, 12:34 AM] flunn, ipre Planned Maintenance

flunn (www123) and ipre (www181) will be taken down shortly to begin drive upgrades. There will be 2 breif periods of downtime for each, at the beginning and end of the upgrades, which should be no more then 10 minutes. An additional notice will be posted at the completion of the upgrades.

[Oct 10, 2002, 11:52 PM] emancholl, glikk, mini, lauta Planned Maintenance

emancholl, glikk, mini, lauta were upgraded this evening to Pentium 4 1.8 Ghz systems as a part of ongoing system upgrades to improve performance. Downtime for each was less than 10 minutes.

[Oct 10, 2002, 12:05 PM] umbar Emergency Maintenance

After showing system instability this morning, umbar has been swapped to a new P4 1.8Ghz chasis as an emergency upgrade. Downtime was about 15 minutes.

[Oct 8, 2002, 11:09 AM] or Downtime

or crashed and was rebooted. Total downtime was less than 10 minutes.

[Oct 8, 2002, 9:58 AM] foma Downtime

foma(www176) crashed and was rebooted. Downtime was under 10 minutes.

[Oct 7, 2002, 11:04 PM] waitak Downtime

Waitak (www283) crashed under high load and had to be rebooted. Total downtime about 10 minutes.

[Oct 7, 2002, 10:16 PM] bhoth Downtime

bhoth (www163) was rebooted to clear up a problem with its network card. Total downtime was under five minutes.

[Oct 6, 2002, 3:12 AM] mini Downtime

mini (www146) crashed under heavy load and had to be rebooted. Downtime was about 10 minutes.

[Oct 5, 2002, 8:39 PM] foma Downtime

foma (www176) crashed under heavy load and was rebooted. Total downtime was about 10 minutes.

[Oct 5, 2002, 6:03 PM] emancholl Downtime

Server was down for approximately 10 minutes and was rebooted.

[Oct 5, 2002, 11:37 AM] quesse Downtime

quesse (www44) was rebooted to clear up a problem with its network card. Total downtime was under 5 minutes.

[Oct 4, 2002, 3:38 PM] onn Downtime

After several unexplained reboots, onn was swapped into a new chassis. Total downtime was approximately 40 minutes.

[Oct 4, 2002, 1:17 PM] onn Downtime

onn (www29) crashed and was rebooted. Downtime was approximately 10 minutes.

[Oct 4, 2002, 1:21 AM] mini Downtime

mini (www126) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes.

[Oct 3, 2002, 11:05 AM] UUnet Outages

Multiple independent sources are reporting massive nationwide outages and delays on the UUnet network. Although we are no longer a UUnet customer, this problem will affect all UUnet-connected networks, and is beyond our control. If you are reaching us on a UUnet path, please contact your ISP for updates on the situation.

[Oct 3, 2002, 10:24 AM] torbaane Downtime

torbaane (www262) was rebooted to clear up a problematic BIOS setting. Downtime was approximately one minute.

[Oct 2, 2002, 10:02 AM] umbar Downtime

umbar(www46) crashed and was rebooted. New RAM needed to be swaped in to help bring the server back online. Total downtime was under 20 minutes.

[Oct 2, 2002, 8:56 AM] kanat Downtime

kanat(www128) became unresponsive and needed to be rebooted. Downtime was under 15 minutes.

[Oct 1, 2002, 12:23 PM] Network Problems

We are continuing to see unacceptably high latency in AT&T's network. We are working with their engineers until this problem is resolved. We may choose to reroute some customer traffic.

[Oct 1, 2002, 10:06 AM] Network Problems Resolved

Our AT&T connectivity has returned to normal, apparently after AT&T rebooted a misbehaving router. We have not been given the full details yet, but no further problems are expected. AT&T reports that they are planning to upgrade the router in question.

[Oct 1, 2002, 9:42 AM] Network Problems

We are seeing major failures on AT&T's network in the past five minutes, causing approximately 10% of our outbound traffic to fail. We are working to reroute traffic, and are contacting AT&T regarding the outage.

[Sep 28, 2002, 1:55 PM] emancholl Downtime

emancholl (www37) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Sep 27, 2002, 6:36 AM] poict Maintenance Completed

The drive swap for poict has been completed. It now has a 60GB drive. Total downtime for the swap was less then 15 minutes.

[Sep 26, 2002, 11:47 PM] ando Maintenance Completed

After the RAM swap on ando (www45), the server crashed repeatedly. The RAM was replaced with a different model, and the server is again running normally. Total downtime was approximately 30 minutes due to this problem.

[Sep 26, 2002, 11:16 PM] poict Planned Maintenance

poict (www195) will be taken down shortly to begin a drive upgrade. Downtime for the start and completion should be less then 10 minutes each. An additional notice will be posted when the swap is completed.

[Sep 26, 2002, 10:58 PM] Network Problems Resolved

Earlier this evening, we discovered problems reaching certain networks through our AT&T OC-3c circuit. After working with AT&T engineers this evening, they have isolated the problem and corrected it. We do not expect any further recurrences of this problem.

[Sep 26, 2002, 9:34 PM] Server Upgrades

This evening, the following servers will be taken down briefly for upgrades to system memory:

• ando
• ampa
• quan

Downtime should be brief, at less than 10 minutes.

[Sep 26, 2002, 4:50 PM] Earthlink Problems Resolved

Although we still have not heard from Earthlink directly, it appears the block on our mail servers has been lifted. If customers experience any further problems with mail deliveries to Earthlink, please contact abuse@pair.com with any bounce messages you may have received.

[Sep 26, 2002, 4:28 PM] Spam Attack

We are currently the victim of a "Joe Job" Spam attack, where thousands of Spams are being sent out that pretend to originate from our network. Naturally, these messages are not originating here, and we are working on action against the perpetrators. Further details are available at http://www.pair.com/pair/support/notices/spam.html

[Sep 26, 2002, 1:13 PM] Earthlink Mail Problems

We have recently discovered that Earthlink is blocking mail from pair.com servers, possibly in response to a recent spam attack which falsely names a customer of ours but does not actually originate from our system in any way. We have attempted to contact Earthlink about a resolution to this issue, but have not yet received any response. Please be advised that ay e-mail originating from pair.com servers sent to Earthlink addresses may not reach its destination. We will post more information once this issue has been resolved.

[Sep 25, 2002, 5:14 PM] Network Problems

Allegiance Telecom has advised us that they are having significant network problems in the Cleveland/Pittsburgh area, and that until they can resolve the matter, we should take down our connectivity to them. We have therefore taken down our DS-3 to Allegiance. All customer traffic is being carried by alternate providers until further notice. Performance will not be impacted.

[Sep 25, 2002, 11:04 AM] merau Mail Delivery

merau (www245) has been under Spam attack for the past 24 hours. We are actively working to block this, during which time e-mail delivery may be delayed for other users on this server.

[Sep 25, 2002, 9:01 AM] Swish-E Updates

We have updated Swish-E on user servers to version 2.2, based on evidence that the previous 2.x version was not working correctly. Before the FreeBSD 4.6-STABLE upgrade, Swish-E 1.3 was in use; this binary is now available at /usr/local/bin/swish-e-13 for backwards compatibility.

We have also installed the SWISHE Perl module for accessing Swish-E 2.2 functionality.

[Sep 25, 2002, 9:00 AM] Network Problems

We are observing poor performance on our DS-3 with Allegiance Telecom. We will be following up with Allegiance today to determine the cause and resolution for this problem; if necessary, we will disable this circuit until the matter can be resolved. Allegiance carries less than five percent of our traffic.

[Sep 25, 2002, 3:56 AM] ynilo, sallane Maintenance Completed

Both drive swaps have been completed. They both have 60GB drives, total downtime was about 15 minutes for each.

[Sep 24, 2002, 10:53 PM] ynilo, sallane Planned Maintenance

ynilo (www166) and sallane (www263) will be taken down shortly to start drive upgrades. There will be 2 breif periods of dowtime, to start and complete the swaps. Each should be less then 10 minutes.

[Sep 23, 2002, 5:06 PM] epsilon Downtime

epsilon (www3) experienced a resource problem which required a reboot. Downtime was approximately 1 minute.

[Sep 23, 2002, 1:33 PM] xi Downtime

xi(www8) became unresponsive and required a file systems check before rebooting. Downtime was under 20 minutes.

[Sep 18, 2002, 10:05 AM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

theta (www4)
mu (www7)
or (www34)
ur (www30)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

[Sep 17, 2002, 2:59 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

ceirt (www39)
uilen (www35)
ando (www45)
thuule (www49)
nuin (www18)
huath (www19)
chi (www14)
omicron (www9)
tinne (www21)
coll (www22)
muin (www23)
gort (www24)
dair (www20)
ruis (www27)
onn (www29)
edad (www31)
ngetal (www25)
ebad (www33)
gamma (www2)
iota (www5)
idad (www32)
kappa (www6)
xi (www8)
rho (www11)
upsilon (www13)
ilwe (www81)
kirlian (www110)
othala (www150)
sothor (www182)
zhiri (www290)
tousko (www291)
gamsi (www289)
gatkasa (www288)
nadakh (www287)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The following upgrades are ongoing, and will be completed tomorrow morning: theta (www4)

[Sep 17, 2002, 2:32 PM] relay.pair.com delays

Relay.pair.com was flooded by spam around 11am. This caused a delay in mail delivery while the spam was removed from the mail queue. Normal email delivery through relay should be back to normal in a few hours. We have taken the necessary steps to prevent this from happening again. We apologize for any inconvenience this may have caused you.

[Sep 17, 2002, 12:32 PM] thuule Maintenance Completed

thuule (www49) had its problems corrected and was returned to normal service without needing a chassis swap. Downtime was approximately five minutes.

[Sep 17, 2002, 12:10 PM] thuule Emergency Maintenance

thuule (www49) has developed hardware issues which will require a chassis swap on an urgent basis. Downtime should be no more than 10 minutes. We will post an additional notice when more information is available.

[Sep 17, 2002, 11:31 AM] bemnet Chassis Upgrade

In an attempt to fix the rebooting issues on bemnet (www158), yesterday we swapped it to a new chassis. It is now a Pentium 4 1.6GHz with 256MB of RAM. We believe this will clear up the rebooting problem on this server.

[Sep 16, 2002, 3:22 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

halla (www68)
beeoro (www69)
tama (www70)
hyarmen (www66)
fearn (www40)
lissa (www72)
ifin (www36)
dyyme (www71)
anca (www53)
eite (www38)
tinco (www41)
parma (www42)
emancholl (www37)
omega (www16)
quesse (www44)
umbar (www46)
anga (www47)
calma (www43)
straif (www26)
harma (www50)
hwesta (www51)
ampa (www52)
ungwe (www48)
ailm (www28)
psi (www15)
beith (www17)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The following upgrades are ongoing, and will be completed tomorrow morning: ceirt (www39)
uilen (www35)
ando (www45)
thuule (www49)

[Sep 16, 2002, 8:24 AM] tau Downtime

tau(www12) crashed under heavy load and had to be rebooted. Total downtime was less than 5 minutes.

[Sep 15, 2002, 10:51 PM] kappa Downtime

Kappa (www6) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes.

[Sep 15, 2002, 7:18 PM] bemnet Downtime

bemnet(www158) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes.

[Sep 14, 2002, 12:28 AM] bemnet Downtime

bemnet(www158) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes.

[Sep 13, 2002, 1:21 PM] ruis Downtime

ruis (www27) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Sep 13, 2002, 9:59 AM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

neter (www132)
aaze (www65)
yanta (www67)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

[Sep 12, 2002, 3:52 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

kayan (www133)
enek (www130)
sether (www95)
gnaaste (www79)
ulwar (www82)
ydhu (www83)
dhaame (www77)
scire (www78)
eeoth (www85)
auma (www86)
mildh (www87)
shen (www88)
aedde (www84)
gao (www111)
zais (www89)
nish (www92)
vuae (www93)
unque (www54)
nuumen (www55)
enda (www80)
noldo (www56)
nwalme (www57)
vala (www59)
vilya (www60)
flaid (www91)
roomen (www61)
arda (www62)
lambe (www63)
silme (www64)
cele (www73)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The following upgrades are ongoing, and will be completed tomorrow morning:

neter (www132)
aaze (www65)
yanta (www67)

[Sep 12, 2002, 1:19 PM] tenshian Downtime

tenshian(www103) crashed and was rebooted. Downtime was under 10 minutes.

[Sep 12, 2002, 12:49 AM] derba Maintenance Completed

The chassis swap for derba has been completed. It is now a Pentium4 1.8GHz, with 512MB of RAM. Total downtime was about 10 minutes.

[Sep 11, 2002, 11:58 PM] derba Planned Maintenance

derba (www104) will be taken down shortly for a chassis swap. Downtime for this should be under 10 minutes. An additional notice will be posted when the swap is completed.

[Sep 11, 2002, 3:30 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

sowilu (www145)
kfin (www151)
yekk (www124)
bemnet (www158)
cuzea (www94)
lepen (www129)
otok (www131)
thurisaz (www136)
falku (www98)
thesel (www99)
aster (www96)
pyyl (www97)
laguz (www148)
tenshian (www103)
derba (www104)
paat (www100)
bellat (www101)
vert (www102)
ingwaz (www149)
reit (www108)
wawrra (www109)
dhabat (www105)
zirx (www106)
zhun (www107)
glikk (www119)
gwind (www74)
flure (www75)
wredhor (www76)
khurla (www112)
maor (www113)
quan (www120)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The following upgrades are ongoing, and will be completed tomorrow morning:

kayan (www133)
enek (www130)
sether (www95)

[Sep 11, 2002, 12:21 PM] Network Maintenance Completed

This morning we have turned up new OC-3c circuits to Genuity and Global Crossing. For Global Crossing, this replaces our existing DS-3 service to their network. For Genuity, we will maintain the existing DS-3 service in parallel for the remainder of its contract (early 2003).

These upgrades expand our total paid transit capacity to more than 900 megabits per second, with connectivity provided by eight different backbone networks. We have further upgrades planned, including an OC-12c circuit (622Mbps total capacity) to AT&T.

We will be making minor routing adjustments in the next few days to rebalance traffic and select optimal destinations for certain networks.

Thank you for choosing pair Networks!

[Sep 11, 2002, 9:45 AM] shema Maintenance Completed

We have completed the drive swap on shema (www203). Total downtime for this maintenance was under 10 minutes.

[Sep 10, 2002, 11:10 PM] Important FreeBSD 4.6-STABLE Updates

Earlier this evening, we deployed a new build of Apache with PHP 4.2.2, with the memory_limit feature disabled, as it was causing problems for many customer scripts. It seems likely that at least one library upon which PHP relies is leaking memory, causing the memory_limit to be tripped randomly by scripts containing certain operations. Any user who uses our PHP4 and has experienced incomplete pages, empty pages, or similar errors, should no longer have these problems.

We have also deployed minor fixes to checkbot, ping, tset, uuencode, and several Perl modules. We have several other fixups and enhancements planned, and are working continuously to ensure the best possible results from this upgrade. We appreciate any reports of problems or improvements at urgent@pair.com.

[Sep 10, 2002, 10:48 PM] shema Emergency Maintenance

shema (www203) will be taken down shortly to begin an emegency drive swap. There will be 2 brief periods of downtime, at the begining and end of the swap. An additional notice will be posted when the swap is completed.

[Sep 10, 2002, 4:59 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

jera (www142)
kenaz (www138)
gebo (www139)
wunjo (www140)
naudhiz (www141)
emra (www183)
berkano (www147)
perthro (www143)
elhaz (www144)
tiwaz (www146)
agho (www187)
ngatwo (www115)
xrra (www116)
enso (www152)
naam (www114)
zatz (www122)
flunn (www123)
yuzz (www117)
humph (www118)
thnad (www121)
ynilo (www166)
neled (www127)
kanat (www128)
vroo (www125)
mini (www126)
jarre (www172)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html.

Upgrades on the following servers are ongoing and will be completed tomorrow morning:

sowilu (www145)
kfin (www151)
yekk (www124)
bemnet (www158)

[Sep 10, 2002, 12:09 PM] Network Maintenance

We will be doing some adjustments to routing involving Genuity and Global Crossing today, in preparation for the activation of OC-3c circuits to both of those networks. No significant disruption of customer traffic is expected; in the event any circuit is down, alternate paths will carry customer traffic.

[Sep 9, 2002, 3:36 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

misilay (www282)
wanote (www284)
auddo (www253)
nandirse (www255)
tuzuba (www256)
mirozo (www257)
mhoh (www191)
athani (www157)
paell (www153)
voot (www154)
haebrath (www156)
diviy (www217)
bhoth (www163)
uumor (www159)
tebba (www160)
ciot (www161)
rult (www162)
vepar (www201)
xomi (www169)
omsano (www164)
taatu (www165)
shema (www203)
ansuz (www137)
dotya (www170)
raitax (www171)
fehu (www134)
uruz (www135)
delsi (www206)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The upgrade on qaske (www250) is ongoing, and will be completed tomorrow.

[Sep 9, 2002, 2:05 PM] ailm Downtime

ailm(www28) became unresponsive and was rebooted. Downtime was less than 10 minutes.

[Sep 8, 2002, 10:05 PM] foma Downtime

foma (www176) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Sep 8, 2002, 1:58 AM] iota Downtime

iota crashed earlier, and was showing signs of a problem with the cpu. It was swapped into a new chassis, and is now an Athlon XP 1600 with 1GB of RAM. Total downtime for the reboot and chassis swap was about 20 minutes.

[Sep 7, 2002, 10:30 PM] emancholl Downtime

emancholl (www37) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Sep 6, 2002, 11:57 AM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

aerre (www185)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

[Sep 6, 2002, 11:46 AM] Network Problems

Our OC-3c circuit to AT&T has bounced twice this morning. We have asked them for an investigation and explanation. Each downtime has lasted less than one minute, and has caused brief routing instabilities while traffic moves to alternate providers.

We will post further information if it becomes available.

[Sep 6, 2002, 3:17 AM] torbaane Downtime

torbaane(www262) crashed after running out of swap space, and had to be rebooted. Total downtime was about 10 minutes.

[Sep 5, 2002, 5:07 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

emlis (www216)
klesh (www200)
lauta (www260)
maym (www202)
maborym (www204)
orvni (www205)
sasi (www207)
imtis (www208)
sallane (www263)
oomur (www209)
amse (www210)
akket (www211)
vsael (www173)
biont (www174)
saakal (www264)
zulle (www175)
foma (www176)
londaste (www177)
gief (www178)
whury (www179)
hauda (www234)
merull (www180)
ipre (www181)
linato (www186)
khin (www184)
zelaza (www247)
aveu (www192)
fiann (www188)
ilceille (www189)
csor (www190)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The upgrade on aerre (www185) is ongoing, and will be completed tomorrow.

[Sep 5, 2002, 9:10 AM] pi Maintenance Completed

We have completed the drive upgrade on pi (www10). Total downtime for this maintenance and the FreeBSD 4.6 upgrade was under 10 minutes.

[Sep 5, 2002, 1:29 AM] FreeBSD 4.6-STABLE Updates

The latest notes on the FreeBSD 4.6-STABLE upgrades have been posted at http://www.pair.com/pair/support/notices/46-upgrade.html and should be examined by any customer who encounters difficulty with their site after the upgrade.

No major system upgrade can be completed without various problems. We are working continuously to resolve any outstanding issues, and document issues which may affect other customers. We appreciate your patience as we work to provide the best possible hosting experience for our customers.

[Sep 4, 2002, 4:11 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

epsilon (www3)
merau (www245)
kodh (www90)
pi (www10)
erraj (www213)
ezera (www281)
naih (www214)
diaf (www215)
oynos (www218)
waitak (www283)
duwo (www219)
penkwe (www220)
ainaz (www221)
fithwor (www222)
fimfi (www223)
sibum (www224)
siyazi (www251)
niwun (www225)
tehun (www226)
eahta (www227)
uinus (www228)
petora (www229)
yitiya (www252)
xebte (www230)
dieci (www231)
virre (www193)
isris (www194)
poict (www195)
nyartu (www254)
namt (www196)
allatu (www197)
morx (www198)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

The upgrade for emlis (www216) is ongoing, and will be completed tomorrow morning.

[Sep 4, 2002, 3:28 PM] epsilon Maintenance Completed

We have completed the drive swap on epsilon (www3). Total downtime for this maintenance and the FreeBSD 4.6 upgrade was under 10 minutes.

[Sep 4, 2002, 3:14 PM] kodh Maintenance Completed

We have completed the drive swap on kodh (www90). Total downtime for this maintenance and the FreeBSD 4.6 upgrade was under 10 minutes.

[Sep 4, 2002, 10:13 AM] epsilon, pi, kodh Drive Swaps

We have completed the FreeBSD 4.6 upgrades on epsilon (www3), pi (www10), and kodh (www90), and have initiated drive swaps on all three of them. These servers immediately began performing more efficently with the upgrade to FreeBSD 4.6, and the current drive swaps should alleviate the problems these servers have experienced. There will be an additional brief period of downtime at the end of these swaps. More information will be posted as the swaps progress.

[Sep 3, 2002, 10:23 PM] FreeBSD 4.6 Upgrades

Today the upgrade of patru (www232) was also completed.

We are maintaining the most current available information on caveats and problems related to the 4.6-STABLE upgrade at http://www.pair.com/pair/support/notices/46-upgrade.html. Please refer to this page if you encounter any problems after your server is upgraded. Of particular note, custom builds of php4.cgi may need recompiled if you are accessing MySQL; Zend Optimizer has been disabled due to buggy behavior (this prevents Zend Encoded scripts from working); and PHP3 no longer has mcrypt and other features (please upgrade to PHP4).

We appreciate our customers' patience as we identify and resolve the outstanding issues along this critical upgrade path.

[Sep 3, 2002, 5:19 PM] FreeBSD 4.6 Upgrades

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

sekweta (www258)
saghotta (www259)
jaane (www261)
torbaane (www262)
tisibo (www265)
xoopi (www266)
lehhdi (www267)
gwaleel (www268)
peswar (www233)
okkod (www235)
muyal (www236)
nallav (www237)
seyyav (www238)
iruvul (www239)
rendu (www240)
zortzi (www243)
sogbe (www274)
iski (www242)
xamsa (www244)
murifwae (www279)
yiwen (www246)
aqquz (www248)
wokkil (www249)
kaazh (www212)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html Upgrades are ongoing on the following servers:

epsilon (www3)
pi (www10)
kodh (www90)
merau (www245)

Upgrades on these servers will be completed tomorrow morning, Wednesday, September 4. At that time, we will also begin drive swaps on epsilon (www3), pi (www10), and kodh (www90).

[Sep 3, 2002, 3:50 PM] Problems with Zend Optimizer

We have discovered that the new version of Zend Accelerator/Optimizer required to interface with PHP 4.2.2, which we have installed on our FreeBSD 4.6-STABLE systems, is buggy. When it is enabled, features such as MySQL access from PHP4 scripts do not work reliably. This has broken many customer scripts, and consequently, we have disabled this module.

The Accelerator/Optimizer is intended only to improve the load time of certain PHP4 scripts. Because it is a closed-source product, we are unable to diagnose or correct the problem it is causing. We note that Zend also sells a product called Encoder, which purports to hide PHP script source, and relies on the Accelerator/Optimizer to interpret the encoded script. We do not support or recommend the Encoder product, and without the Optimizer, Encoded scripts will not work.

We hope to resolve the Zend Accelerator/Optimizer bugs with Zend directly, but have chosen the path that does the least harm to customer functionality by disabling it.

[Aug 31, 2002, 3:31 AM] sekweta Maintenance Completed

The drive swap of sekweta has been completed. We also took this opportunity to upgrade the chassis. It now has an Athalon XP 1600, with 1024MB of RAM. The drive has been upgraded to a 60GB drive. Total downtime was about 25 minutes.

[Aug 30, 2002, 3:38 PM] sekweta Downtime

sekweta (www258) was rebooted for a BIOS update. Total downtime was under 5 minutes. A drive swap will be done on sekweta tonight, which will require a brief period of downtime after the swap is complete. More information will be posted as the swap progresses.

[Aug 30, 2002, 2:34 PM] emancholl Downtime

emancholl (www37) hung and had to be rebooted. Total downtime was under 10 minutes.

[Aug 29, 2002, 4:07 PM] STABLE Upgrades Deferred

During the past twenty-four hours, we have tracked down an obscure bug in PNG creation in PHP4 by way of the GD library. This has finally been resolved, along with other minor fixes and updates. We are now analyzing problem reports by users of FrontPage 2002. Consequently, we will be deferring the servers which were scheduled for 4.6-STABLE upgrades today; they will be rescheduled for Monday, September 9.

Barring any new and unresolved problems, the upgrade schedule will proceed as planned on Tuesday, September 3. Several of the servers scheduled for that day will promptly receive a drive swap, as the new more efficient 4.6-STABLE code makes such a swap practical.

[Aug 29, 2002, 1:05 PM] kappa Downtime

kappa(www6) crashed and required an extensive file system check before rebooting. Downtime was approximately 30 minutes.

[Aug 29, 2002, 4:14 AM] straif Maintenance Completed

The drive swap on straif has been completed. It now has a 60GB drive. Total downtime for this was about 15 minutes.

[Aug 29, 2002, 3:32 AM] shen Downtime

shen(www88) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes.

[Aug 29, 2002, 12:48 AM] sekweta drive update

For unforseen reasons, we are going to delay the swap for sekweta. An additional notice will be posted when the swap is actually started.

[Aug 28, 2002, 10:46 PM] straif, sekweta Planned Maintenance

straif(www26) and sekweta(www258) will be taken down shortly to begin drive upgrades. There will be 2 breif periods of downtime, about 10 minutes each. 1 to start the drive upgrade, and another to complete it.

[Aug 28, 2002, 5:17 PM] omicron Downtime

omicron (www9) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Aug 28, 2002, 11:05 AM] FreeBSD 4.6-STABLE Maintenance

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

gessire (www273)
dsumaui (www275)
orroka (www276)
wutare (www277)
midhae (www278)
teghis (www280)
qenni (www168)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http ://www.pair.com/pair/support/notices/46-upgrade-schedule.html

[Aug 28, 2002, 8:18 AM] enso Downtime

enso was became unresponsive and was rebooted. Downtime was under 10 minutes.

[Aug 28, 2002, 12:36 AM] epsilon Slow Mail Delivery

We are aware of continuing slow e-mail delivery on epsilon (www3), and have taken steps this evening to alleviate these delays. We are also accelerating the planned upgrade to FreeBSD 4.6-STABLE for this and several other servers, as the updated operating system provides greatly improved disk performance. Based on the results of 4.6-STABLE upgrades this week, we expect to schedule and complete the upgrade for epsilon and other servers early next week.

We appreciate your patience as we strive to resolve this problem.

[Aug 27, 2002, 10:30 PM] naudhiz Downtime

naudhiz (www141) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Aug 27, 2002, 1:16 PM] glavandi, thenge, terfideu, margum Maintenance Completed

The following servers have been upgraded to FreeBSD 4.6-STABLE today:

glavandi (www269)
thenge (www270)
terfideu (www271)
margum (www272)

If you encounter any difficulties as a result of this upgrade, please contact us at urgent@pair.com immediately. Details of this upgrade are available at http://www.pair.com/pair/support/notices/46-upgrade.html and the schedule is published at http://www.pair.com/pair/support/notices/46-upgrade-schedule.html

[Aug 27, 2002, 3:36 AM] pi Downtime

pi (www10) became unresponsive and required a reboot. Total downtime was about 10 minutes.

[Aug 27, 2002, 2:21 AM] neter Maintenance Completed

neter was taken down breifly to complete the drive swap. Downtime was about 10 minutes.

[Aug 26, 2002, 5:26 PM] foma Downtime

foma (www176) crashed and was rebooted. Downtime was around 10 minutes.

[Aug 26, 2002, 2:24 PM] FreeBSD STABLE Upgrades

The schedule for this week's FreeBSD 4.6-STABLE upgrade has been rearranged slightly, delaying two servers from Monday to Tuesday. The upgrades are working correctly, and no further disruptions of the schedule are expected.

[Aug 26, 2002, 11:08 AM] emancholl Downtime

emancholl(www37) crashed and was rebooted. Downtime was around 10 minutes.

[Aug 25, 2002, 2:01 PM] emancholl Downtime

emancholl (www37) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Aug 25, 2002, 1:02 PM] iota Downtime

iota (www5) was down for approximately 15 minutes this afternoon. Extreme server slowness was eventually attributed to a bad chassis fan, which has been replaced. The server should again be functioning normally.

[Aug 24, 2002, 7:14 PM] beeoro Emergency Maintenance

After monitoring beeoro a problem occurred which may have been indicative of a hardware problem. A preemptive chassis swap was down and now beeoro has been changed to a Pentium III 1000, with 512Mb of RAM.

[Aug 24, 2002, 6:38 PM] beeoro Downtime

beeoro was down, and as a result was rebooted this evening. Downtime was approximatly 15 minutes, and occurred at 6:20 pm.

[Aug 23, 2002, 11:41 AM] neter Maintenance Completed

All user files on neter have copied over to the new drive at this time, and neter is running on the new drive. We are leaving the old drive mounted as a secondary for now, in case we find any remaining files which need to be recovered. The drive will be removed during an evening maintenance window at a later date once we are sure all is well.

[Aug 23, 2002, 5:44 AM] tinne Maintenance Completed

The drive swap for tinne has been completed. It now has a 60GB drive. Total downtime was about 10 minutes.

[Aug 23, 2002, 4:37 AM] paell Maintenance Completed

The drive swap for paell has been completed. It now has a 60GB drive. Total downtime was about 10 minutes.

[Aug 23, 2002, 12:25 AM] neter Maintenance Continued

While in the middle of a drive swap, neter (www132) crashed and treid to reboot. This obviously complicted the already existing swap, and the server was down for approximately 30 minutes. It is back up now, and the drive swap has resumed. We will post a notice once this is completed.

[Aug 22, 2002, 2:02 PM] chi Downtime

chi(www14) became unresponsive and was rebooted. Downtime was approximately 10 minutes.

[Aug 22, 2002, 1:48 PM] tinne, paell Planned Maintenance

We have observed warning conditions on the hard drives in tinne (www21) and paell (www153) which may indicate an imminent failure. Consequently, this evening after 6pm Eastern time, we will be taking these servers offline to initiate drive swaps. There will be two brief periods of downtime on each server, less than ten minutes each. More information will be posted as it becomes available.

[Aug 22, 2002, 11:40 AM] vert Downtime

vert(www102) crashed and was rebooted. Downtime was under 10 minutes.

[Aug 21, 2002, 10:42 PM] neter Emergency Maintenance

neter (www132) was down this evening for 15 minutes. Upon returning to service, it was noticed that there were drive errors that necessitate an immediate drive replacement. We'll be starting the exchange momentarily: it will entail two periods of downtime, each of which should last no longer than 10 minutes.

[Aug 21, 2002, 7:42 PM] epsilon Planned Maintenance

Epsilon was taken down briefly this evening for a server upgrade to a P4 1.8Ghz processor. Downtime was less than 5 minutes.

[Aug 21, 2002, 7:41 PM] ilwe, kirlian, othala, sothor Maintenance Completed

The drive upgrades on these servers has been completed at this time. Additionally, kirlian and othala were upgraded to new P4 chasis.

[Aug 21, 2002, 9:08 AM] aqquz Downtime

aqquz(www248) became unresponsive and was rebooted. Total downtime was under 10 minutes.

[Aug 21, 2002, 12:07 AM] torbaane Downtime

torbaane(www262) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes

[Aug 20, 2002, 7:04 PM] FreeBSD 4.6-STABLE Upgrades

We will begin upgrading our user servers to FreeBSD 4.6-STABLE on Monday, August 26. This has been delayed by several days due to last-minute improvements and further testing. The process is expected to be painless, quick, and the project will be complete soon after it begins.

Full details are available at http://www.pair.com/pair/support/notices/46-upgrade.html

[Aug 20, 2002, 4:38 PM] Server Upgrades

The following servers will be taken down briefly this evening to begin drive swaps to upgrade available drive capacity:

• ilwe
• kirlian
• othala
• sothor

Another notice will be posted once work has been completed.

[Aug 20, 2002, 3:50 PM] iski Downtime

iski(www242) was rebooted to clear up an internal error. Downtime was under 5 minutes.

[Aug 20, 2002, 2:00 PM] naudhiz Downtime

naudhiz(www141) become unresponsive and was rebooted. Downtime was under 10 minutes.

[Aug 18, 2002, 7:58 PM] neter Downtime

neter was down for approximately 15 minutes this evening at around 7:26pm.

[Aug 16, 2002, 3:51 AM] pi Downtime

pi(www10) became unresponsive under high load, and had to be rebooted. Downtime was about 20 minutes. We are closely monitoring this server for any additional causes of high load.

[Aug 15, 2002, 4:03 PM] Network Problems

Since around 3:00 PM Eastern Time, our DS-3 To Allegiance Telecom (formerly Intermedia/Digex) has been out of service. We've been advised that they are having "major routing issues" on much of the east coast. Traffic will be carried over our seven other providers during this outage.

[Aug 15, 2002, 11:20 AM] isris Downtime

isris (www194) experienced severa problems with a customer script which unfortunately was very difficult to track. The server has been returned to normal operations at this time. Downtime was approximately 30 minutes.

[Aug 14, 2002, 8:06 PM] sasi Maintenance Completed

The drive swap on sasi (www207) has been completed. Total downtime between both reboots was approximately 45 minutes.

[Aug 14, 2002, 5:44 AM] epsilon Maintenance Completed

The drive swap on epsilon(www3) has been completed. It now has a 60GB drive. Total downtime was about 15 minutes.

[Aug 14, 2002, 4:59 AM] gwaleel Maintenance Completed

The drive swap on gwaleel has been completed. Total downtime was about 10 minutes.

[Aug 13, 2002, 11:31 PM] sasi Emergency Maintenance

The problem on server sasi (www207) has in fact been isolated to a drive problem. As a result we have been a drive swap. The server is now up, and should remain up throughout the course of the swap. It will be taken back down again once the swap is completed.

[Aug 13, 2002, 11:20 PM] gwaleel Emergency Maintenance

We have detected a failing drive on gwaleel(www268), and will be starting a drive swap. There should be 2 breif periods of down time, at the start and to complete the swap.

[Aug 13, 2002, 10:03 PM] sasi Emergency Maintenance

sasi (www207), in the course of investigating performance issues, was found to have a malfunctioning fan. During the chassis swap the drive started to show signs of failure. A drive swap has been intiated to correct the problem. More details to follow.

[Aug 13, 2002, 3:40 PM] delsi Downtime

delsi(www206) crashed due to high load and was rebooted. Downtime was under 15 minutes.

[Aug 12, 2002, 11:10 PM] epsilon Emergency Maintenance

Drive problems have been discovered on epsilon (www3), and as a result we are beginning a hard drive swap. There should be two brief periods of downtime during this swap: one now, and one when the swap is completed.

[Aug 12, 2002, 8:31 AM] enek Downtime

enek(www130) became unresponsive and was rebooted. Downtime was under 15 minutes.

[Aug 12, 2002, 4:42 AM] pi Downtime

pi (www10) stopped responding. Upon rebooting, the server required a manual disk check. The disks were cleaned and the server was brought back up. Total downtime was about 30 minutes.

[Aug 12, 2002, 3:04 AM] nish Maintenance Completed

The chassis swap for nish (www92) has been completed.

[Aug 12, 2002, 2:47 AM] nish Emergency Maintenance

nish (www92) has crashed several times and is not booting. We are currently performing a chassis swap. Downtime should be under 20 minutes.

[Aug 11, 2002, 12:41 PM] morx Downtime

morx (www198) was rebooted to clear an issue with the network interface card. Downtime was under 10 minutes.

[Aug 9, 2002, 8:48 AM] pi Downtime

pi (www10) became unresponsive this morning and was rebooted. Downtime was under 10 minutes.

[Aug 8, 2002, 1:59 AM] Network Problems Resolved

After almost exactly 25 hours of downtime, our DS-3 circuit to Allegiance Telecom (Intermedia) is once again active. We have not yet received any consistent or complete explanation of the outage, although the scenario they described throughout the day of a "global network outage" seems unlikely at best.

[Aug 7, 2002, 5:09 PM] FreeBSD 4.6-STABLE Upgrades

We have officially announced our plans to upgrade all servers to FreeBSD 4.6-STABLE. These upgrades will begin on August 19, 2002, and further information is available at http://www.pair.com/pair/support/notices/46-upgrade.html

[Aug 7, 2002, 3:20 PM] Network Maintenance

Allegiance Telecom, operators of the Intermedia network (formerly part of Digex), have advised us that after their overnight maintenance window, the majority of their global network is out of service, and that there is still no estimated time to repair.

As we mentioned before, our diverse network has ample capacity to accommodate unusual and extended outages; therefore, this has not affected our performance. We will seek and post further news this evening.

[Aug 7, 2002, 11:16 AM] Network Maintenance

Beginning shortly after midnight last night (Eastern time), Allegiance Telecom, who currently operate the Intermedia network, began one of several major schedule maintenance windows they have recently announced. Although the maintenance was supposed to be complete by 8am Eastern time, as of 11:15am today, our DS-3 service to their network has not been restored, and they have provided no estimate as to when that service will be restored.

Of course, our network is running with no difficulty, as we utilize eight diverse providers. We will post an update when this circuit becomes available again, but in the meantime, customers and site visitors should experience no reduction in performance.

[Aug 7, 2002, 10:58 AM] pi Downtime

pi (www10) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Aug 5, 2002, 7:53 PM] diaf Maintenance Completed

We have completed the chassis swap of diaf (www215). It is now an Athlon XP 1600 with 1GB of RAM. Total downtime for this server was under 20 minutes.

[Aug 5, 2002, 7:48 PM] diaf Emergency Maintenance

diaf (www215) crashed, and has refused to boot. We are currently performing a chassis swap. Downtime should be under 15 minutes.

[Aug 5, 2002, 4:55 PM] omega Downtime

omega (www16) became unresponsive and was rebooted. Downtime was under 5 minutes.

[Aug 5, 2002, 4:47 PM] pi Emergency Maintenance

In order to allow pi (www10) to more quickly complete its data swap for the remaining users, we have disabled the periodic processing of user configuration changes. This means that changes made in the Account Control Center will not take effect until this swap is complete, but the upside is that the swap should finish more quickly.

We will restore this to normal operation as soon as the swap is complete.

[Aug 5, 2002, 4:13 PM] pi Emergency Maintenance

For a brief period this afternoon, some user accounts on pi (www10) lost their configuration data as a side effect of the ongoing restore. This was brought to our attention and promptly restored to proper operation.

We are still working to restore the five user accounts which remain on the old drive in pi. As it turns out, these five user accounts are among nearly a dozen which contain an amazing number of files in single directories.

As files are added to a directory, the size of that directory grows, and at a certain point, kernel performance accessing that directory drops dramatically, to the detriment of other filesystem activity, as well. The current swap of user data has been brought to a near standstill by the presence of dozens of directories, in some cases containing as many as one hundred thousand files!

This performance limitation is greatly reduced in FreeBSD 4.6-STABLE with the new UFS_DIRHASH code. Our system upgrades to FreeBSD 4.6-STABLE will begin this month; in the last week, we have made several final security updates to that codebase.

We will continue to pursue the swap of these user accounts, periodically pausing in order to allow the e-mail queue to proceed. As this server has quite a few busy sites, we will be moving user accounts from it as soon as the current crisis has passed, and also upgrading the chassis to our latest model.

Any user having difficulty with their sites or account on pi should contact us for further assistance. We apologize for the ongoing trouble with this server and appreciate your patience.

[Aug 5, 2002, 7:18 AM] pi Emergency Maintenance

Earlier this morning, pi (www10) crashed during the final stages of its restoration to a new drive. Only five user accounts remain affected, and we are working to manually restore these accounts from the old, damaged drive.

We have noted that pi is unusual in that it contains over one million user files totalling approximately twenty gigabytes. The sheer number of files, in some cases stored in huge directories, is the contributing factor which has caused some parts of this restore to go slowly.

As before, no user accounts should be out of service at this time.

[Aug 5, 2002, 3:36 AM] pi Downtime

In the process of finishing the final stages of the swap, pi once again crashed. We are now going to finish the swap, and then investigate thoroughly the reasons for the crashes in the next few days. It is back up, and will be monitored closely over the night.

[Aug 4, 2002, 12:09 PM] pi Emergency Maintenance

Several accounts have been identified which were only partly restored from the old drive; these accounts have been reset to use symlinks to old data until the new copy is prepared. At this time, we believe all customer sites are running on pi(www10), and at any given time, no more than one account should be in the midst of a copy, for minimal downtime.

We appreciate your patience, and will post further information as it becomes available.

[Aug 4, 2002, 10:00 AM] pi Emergency Maintenance

The emergency maintenance on pi (www10) is unfortunately still continuing. On the bright side, we have switched to a more sophisticated restore approach which should not only complete more quickly, but provides users with access to their accounts and Web sites in the interim.

We are improving our systems procedures to better accommodate this unusual failure mode in the future. We extend our most sincere apologies to customers affected by this failure and the resulting downtime. We hope to have pi restored to full service later tonight, and expect that in the meantime, it will largely function correctly.

Again, we apologize for this problem, and thank you for your continued faith and business. Further updates will be posted as information becomes available.

[Aug 3, 2002, 12:58 PM] pi Status Update

The drive swap on pi is still in progress. Unfortunately, due to the condition of the original drive, file restoration is progressing slower than expected. Because of this, we are not able to provide an accurate estimate of when the drive swap will be completed. Web services will be restored as the files are restored. Mail sent to any accounts on pi will remain in the queue until the mailboxes and rules have been properly restored. We thank you for your continuing patience and understanding in this matter.

[Aug 3, 2002, 12:07 PM] theta Downtime

theta (www4) was rebooted to clear an issue with the network interface card. Downtime was under 10 minutes.

[Aug 2, 2002, 6:25 PM] pi Update

The swap on pi continues, slowly. Some accounts have only been partially restored, which is not cause for alarm. All accounts will be fully restored once work has been completed. We are not sure how long this swap could take, it may yet be several hours. However, our staff will be monitoring its progress until completion.

[Aug 2, 2002, 1:53 PM] pi Status

The drive swap is continuing on pi. As this is an older server with many users with extensive file systems, this has been taking longer than usual. At this stage, over half of user files have been recovered, and the process is accelerating. We hope to have all files recovered within the next couple of hours.

[Aug 2, 2002, 11:15 AM] ifin Downtime

ifin (www36) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Aug 2, 2002, 6:01 AM] pi Emergency Maintenance

pi has web services starting to be run again, some domains might still be calling up the pair default pages while the directories are still being copied over. If you find your account empty still, it'll be repaired soon.

[Aug 2, 2002, 2:00 AM] pi Emergency Maintenance

The main drive for pi was so badly damaged, it was unable to perform any routine diagnostics/fixing; we're still able to get at most of the data, but we have to do this manually with the old drive on secondary; so pi will effectively be down for a while yet.

[Aug 2, 2002, 1:47 AM] pi Emergency Maintenance

We have discovered a warning condition on the hard drive of pi (www10). We will be starting a drive swap at this time. There should be one additional outage to complete the swap, which should be no longer then 10 minutes.

[Aug 1, 2002, 1:08 PM] Security Advisory Update

We have posted new security advisories for Gallery and phpNuke. The posting for phpNuke is a general recommendation to upgrade to version 5.6, which was released on June 4, 2002.

The advisories can be found at http://www.pair.com/pair/support/notices/security.html.

[Jul 29, 2002, 5:05 PM] othala Downtime

othala (www150) was rebooted to clear an issue with the network interface card. Downtime was under 10 minutes.

[Jul 28, 2002, 9:37 AM] Security Notices Launched

Today we have launched a new Security Notices page on our Web site. We have found an increasing trend of customer Web sites being broken into and either defaced or misused, with the attack vector being popular software packages customers have installed for their own use. To help combat this problem, we will post updates to the Security Notices page whenever we become aware of vulnerabilities in the packages we have found popular among our customers.

The new Security Notices page can be found at http://www.pair.com/pair/support/notices/security.html.

[Jul 27, 2002, 6:59 PM] torbaane Downtime

torbaane (www262) was rebooted to clear up an issue with FTP. Total downtime was under 5 minutes.

[Jul 25, 2002, 4:08 PM] Server Upgrades

This evening, the following servers will be taken down briefly for RAM upgrades to improve server performance:

• derba
• yiwen
• chi
• yanta

Downtime should be less than 10 minutes for each server.

[Jul 24, 2002, 3:36 PM] yiwen Downtime

yiwen(www246) became unresponsive and was rebooted. Downtime was under 5 minutes.

[Jul 24, 2002, 11:56 AM] Phorum Vulnerabilities

All users of the Phorum software versions 3.3.2a and earlier are encouraged to upgrade immediately to version 3.3.2c available at phorum.org. Vulnerabilities in earlier versions allow third parties to execute arbitrary commands from your account. This can result in the loss of your files, as was a problem for phorum.org themselves, or your account can be hijacked for illicit activities such as spamming. The current version of Phorum patches these vulnerabilities.

[Jul 23, 2002, 3:55 PM] ynilo Downtime

ynilo(www166) crashed and required an extensive file system check to bring it back up. Approximate downtime was about 15 minutes.

[Jul 23, 2002, 9:04 AM] tisibo Downtime

tisibo(www265) became unresponsive and was rebooted. Downtime was under 10 minutes.

[Jul 20, 2002, 1:17 PM] enek Downtime

enek (www130) was rebooted to clear an issue with the network interface card. Downtime was around 10 minutes.

[Jul 19, 2002, 11:26 PM] neter Maintenance Completed

The drive swap on neter has been completed at this time.

[Jul 19, 2002, 6:34 PM] neter Planned Maintenance

When neter was rebooted, we discovered errors were showing on its drive. As a result, we are proactively swapping the drive before it demonstrates any further problems. This will result in two brief outages while work is completed.

[Jul 19, 2002, 6:15 PM] kodh Maintenance Completed

The drive swap on kodh has been completed. We believe the failing drive had a major impact on server preformance, so this swap should noticably improve the responsiveness of this server.

[Jul 19, 2002, 10:01 AM] neter Downtime

neter(www132) became unresponsive and was rebooted. Downtime was around 10 minutes.

[Jul 19, 2002, 9:27 AM] neter Downtime

neter(www132) rebooted it self to clear up a network issue. Dowtime was under 5 minutes.

[Jul 19, 2002, 8:42 AM] nuin Downtime

nuin(www18) was rebooted this morning after it became unresponsive. Downtime was under 10 minutes.

[Jul 17, 2002, 9:25 PM] kodh Downtime, Maintenance

kodh (www90) was rebooted to clear up a network issue. During this downtime, we have also initiated a drive swap. Warning conditions have appeared on this drive which may indicate an imminent failure. We apologize for the inconveience and appreciate the continued patience of customers on this server while we work to eliminate these issues.

[Jul 17, 2002, 7:16 PM] dotya Downtime

dotya (www170) was rebooted to clear up an issue with FTP. Downtime was under 5 minutes.

[Jul 17, 2002, 2:26 PM] chi Downtime

chi (www14) was rebooted to clear up an issue with swap space. Total downtime was under 10 minutes.

[Jul 17, 2002, 10:59 AM] oynos Downtime

oynos (www218) experienced heavy load and was rebooted. While the server was being rebooted, we upgraded the RAM to 512MB. Downtime was approximately five minutes.

[Jul 17, 2002, 12:31 AM] ciot Downtime

ciot (www161) became unresponsive and had to be rebooted. Total downtime was about 10 minutes.

[Jul 16, 2002, 9:57 PM] foma Downtime

foma (www176) crashed and was rebooted. The total downtime was under 10 minutes.

[Jul 16, 2002, 9:41 PM] bellat Downtime

bellat (www101) became unresponsive and was rebooted. The total downtime was under 10 minutes.

[Jul 16, 2002, 6:07 PM] kodh Performance Issues

We are aware of performance issues on kodh (www90) which are affecting customers on this server. We are pursuing several possible solutions to this matter currently. We apologize for any difficulties in the meantime, and fully expect to have the matter resolved within the very near future. There is the possibility that troubleshooting measures will occur over the next 2-3 days. We will post more information as it becomes available.

[Jul 16, 2002, 10:51 AM] kodh Maintenance Completed

The maintenance on kodh (www90) has been completed. The server was upgraded to 1GB of DDR RAM. We expect this upgrade will greatly help in the sluggish performance displayed by the server. Downtime was approximately 5 minutes.

[Jul 16, 2002, 10:32 AM] kodh Emergency Maintenance

Due to performance problems on kodh (www90), we will be taking the server down on an emergency basis to add RAM. Downtime should be less than 10 minutes. We will post more information as it becomes available.

[Jul 15, 2002, 9:40 PM] paat Downtime

paat (www100) was rebooted after it became unresponsive. Total downtime was under 10 minutes.

[Jul 15, 2002, 4:09 PM] lauta Downtime

lauta(www260) crashed under high load and was rebooted from the console. Downtime was under 15 minutes.

[Jul 15, 2002, 11:12 AM] onn Downtime

onn(www29) crashed under high load and rebooted. Downtime was under 15 minutes.

[Jul 15, 2002, 8:49 AM] Network Problems Resolved

After a record-setting fifty-five hour outage, our DS-3 circuit to Cable and Wireless has been restored to normal service. Their technicians had insisted that we had the line looped on our end, which of course we did not. After requesting that they again open tickets with the telcos providing the line, it was discovered that in fact there was a loop accidentally set at AT&T. That was fixed, and now the circuit is running again.

We are still faced with considerable uncertainty as to whether or not we wish to continue service with Cable and Wireless, but that will be determined based on their ability to rectify this and other problems we have had with the service. As always, we will strive to provide the best possible network connectivity for our customers.

[Jul 14, 2002, 1:04 AM] Network Update

As we've mentioned earlier today, our DS-3 circuit to Cable and Wireless went down around 1:45am Saturday morning. The circuit has remained down for nearly 24 hours. For the first ten to twelve hours, we were advised that an OC-48 to Cleveland had failed and was being worked on by the telco. After this, there were no further updates until around 8pm, when we were suddenly told that the problem must be on our end.

It appears that Cable and Wireless is very fond of the idea that their maintenance and subsequent circuit failure has led to a situation which is somehow our fault. We have no evidence that they are doing any further investigation or troubleshooting on the downed circuit. Our most recent statement from their technician was that "we must have a loose cable". Of course, we do not.

We are advising Cable and Wireless that they need to continue to work on the problem if they wish to continue to have our business. Specifically, if the circuit is not fully operational by 10am Eastern time Monday, absent an adequate technical explanation and estimated time-to-repair, we will terminate the service and expect a full refund, as well as credits under the service-level agreement.

We were pursuing a possible OC-3c upgrade with Cable and Wireless, but we are very disappointed in their handling of this outage, and now feel that unless significant improvements are made, we will purchase no further services from them.

We apologize for any inconvenience to our customers, although we have not measured any negative impact on network performance. Also, as we have sufficient alternate capacity, we do not expect any performance impact if this circuit remains down under weekday traffic loads, or even if it is in fact discontinued permanently.

[Jul 13, 2002, 12:22 PM] iota Downtime

iota (www5) was rebooted to clear an issue with the network interface card. Downtime was less than 5 minutes.

[Jul 13, 2002, 10:48 AM] Network Problems

Our DS-3 to Cable & Wireless has been down since approximately 1:45am Eastern time. This began with a routine maintenance window, but we have since been advised that the maintenance caused other serious problems and that many circuits from Cleveland are down.

C&W reports that the matter has been escalated with their telco provider in Cleveland, and that it should be resolved soon. Customer traffic has been carried by alternate providers with no impact on performance.

[Jul 13, 2002, 10:01 AM] aedde Downtime

aedde (www84) was rebooted after it became unresponsive. Total downtime was under 10 minutes.

[Jul 12, 2002, 3:10 PM] merau Maintenance Completed

The server upgrade to merau has been completed at this time. Downtime was less than 10 minutes.

[Jul 12, 2002, 2:27 PM] merau Planned Maintenance

merau will be taken down briefly today for a server upgrade to improve performance. Another notice will be posted when this is completed.

[Jul 11, 2002, 3:53 PM] merau Downtime

merau (www245) was rebooted when the system became unresponsive. Downtime was under 10 minutes. We are continuing to monitor this server closely.

[Jul 11, 2002, 2:34 PM] foma Downtime

foma (www176) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Jul 11, 2002, 11:29 AM] roomen Downtime

roomen (www61) crashed under a heavy load and was rebooted. The downtime was under 10 minutes.

[Jul 9, 2002, 2:47 PM] merau Downtime

merau(www245) crashed due to high load and was manually rebooted. Downtime was under 10 minutes.

[Jul 9, 2002, 9:08 AM] My pair Account Control Center

The Account Control Center has returned to normal operation. We apologize for the inconvenience.

[Jul 9, 2002, 8:46 AM] My pair Account Control Center

We are currently experiencing a problem with the servers that handle the My pair Account Control Center. Users may experience difficulty logging into the Account Control Center. We are investigating and hope to have the problem resolved shortly.

[Jul 6, 2002, 4:10 AM] ando Maintenance Completed

The chassis swap of ando has been completed. It is now a Pentium 4 1.6Ghz with 512MB of RAM. Total downtime for this was 10 minutes.

[Jul 6, 2002, 3:38 AM] ando Emergency Maintenance

After 2 crashes in the last 30 minutes, we will be swapping the chassis of ando (www45.pair.com). Additional downtime should be less than 10 minutes.

[Jul 5, 2002, 7:54 PM] ainaz Downtime

ainaz (www221) was rebooted to clear up a problem with the netweork interface card. Total downtime was about 10 minutes.

[Jul 5, 2002, 3:13 PM] ando Downtime

ando (www45) crashed under haevy load and required a manual disk check before being brought online. Downtime was 15 minutes.

[Jul 5, 2002, 12:06 AM] ando Maintenance Completed

We have completed the drive swap on ando (www45). Total downtime for this maintenance was 10 minutes.

[Jul 4, 2002, 2:17 PM] ando Emergency Maintenance

We have observed a warning condition on the hard drive in ando (www45) which may indicate an imminent failure. Consequently, we will be taking this server down shortly to initiate a drive swap. There will be two brief periods of downtime, less than ten minutes each. Additional information will be posted as it becomes available.

[Jul 4, 2002, 11:39 AM] Relay mail server downtime

Our outgoing mail server, relay.pair.com experienced problems with its network interface card this morning from approximately 10:00 AM until 11:00 AM EDT. During that time customers may have had difficulty sending outgoing mail. We have rebooted the server and will continue to monitor it closely.

[Jul 3, 2002, 2:41 PM] Network Problems Resolved

The Cable and Wireless circuit returned to normal operation at approximately 2:15 PM Eastern Time. We will continue to monitor it closely.

[Jul 3, 2002, 1:11 PM] Cable and Wireless Update

Cable and Wireless has advised us that they have experienced a router failure in Cleveland, Ohio, causing our circuit, among others, to remain down while they replace failed equipment. The circuit is expected to return to normal operation soon. We will post further information as it becomes available.

[Jul 3, 2002, 12:46 PM] Network Problems

At 12:06pm Eastern time, our DS-3 to Cable and Wireless went down and stayed down. We have a ticket open with the CW NOC, but they have no information available yet on the outage or estimated time to repair. Our traffic is being carried by our other providers with no difficulty.

[Jul 2, 2002, 11:56 AM] thenge Downtime

thenge (www270) crashed due to high load and was rebooted. Downtime was under 15 minutes.

[Jul 2, 2002, 10:43 AM] omega Downtime

omega (www16) crashed due to high load and was rebooted. Downtime was under 10 minutes.

[Jul 1, 2002, 5:06 PM] yiwen Downtime

yiwen (www246) crashed due to high load and was rebooted. Downtime was under 10 minutes.

[Jun 30, 2002, 1:21 PM] lepen Downtime

lepen (www129) was rebooted to clear an issue with the network interface card. Downtime was less than 5 minutes.

[Jun 29, 2002, 1:41 PM] epsilon Downtime

epsilon (www3) became unresponsive due to high load and was rebooted. The total downtime was under 10 minutes.

[Jun 29, 2002, 5:01 AM] lauta Downtime

Lauta should be back up and running. Again, some mail will have been lost, but all else should be functioning as normal.

[Jun 29, 2002, 3:54 AM] lauta Downtime

Lauta is running services now. We were successful in getting the home directory and web directory data from the old drive before it failed entirely, but we were not successful in getting the mail. We are restoring from backups, but mail received in the past 24 hours will have been lost.

[Jun 29, 2002, 2:15 AM] lauta Downtime

We are continuing to work on the server to bring it back. Unfortunately, the drive was too badly damaged to be able to run it while swapping the files over (as usual), so services are down while we are doing that. We are roughly two-thirds of the way through the process at this time.

[Jun 28, 2002, 10:22 PM] lauta Emergency Maintenance

We have detected significant drive errors on lauta (www260). We will be taking the server down at once to swap the hard drive. More information will be posted as it becomes available.

[Jun 28, 2002, 11:53 AM] Network Problems

This morning at 11:22 Eastern time, one of our internal gateway switches failed. This switch, a Foundry BigIron, had similar failures four months ago, but the problem was believed to have been resolved by a workaround. We no longer have confidence in that workaround, and will be taking two approaches to solve the problem. First, we are demanding a complete replacement of the unit by Foundry. Second, we are adding new switching equipment to operate in parallel to the Foundry units, from a different vendor.

This switch carries approximately 30% of our network traffic. During the 20-minute outage, most of that traffic was carried by alternate switches, although the failover method is somewhat imperfect. Overall traffic levels varied between normal and 20% reduced. Some customers may have experienced temporary difficulties reaching their servers and Web sites.

We will post further information on any developments with this problem. We have also prepared for subsequent failures, so that any resulting outage should be five minutes or less.

We apologize for the inconvenience to our customers, and hope that Foundry will take the problem as seriously as we do.

[Jun 28, 2002, 10:49 AM] yiwen Downtime

yiwen(www246) crashed due to high load and was manually rebooted. Downtime was under 5 minutes.

[Jun 27, 2002, 2:14 AM] ngatwo Downtime

ngatwo (www115) was rebooted to clear up a problem with its network card. Total downtime was about 5 minutes.

[Jun 26, 2002, 2:15 PM] ando Downtime

ando (www45) crashed under a high load and was rebooted. The downtime was under 15 minutes.

[Jun 26, 2002, 1:31 PM] yiwen Downtime

yiwen (www246) crashed under a high load and was rebooted. The downtime was under ten minutes.

[Jun 25, 2002, 6:02 PM] epsilon Downtime

epsilon(www3) crashed under high load was manually rebooted. Downtime was under 15 minutes.

[Jun 25, 2002, 4:58 PM] epsilon Downtime

epsilon(www3) crashed under high load and was rebooted from the consol. Downtime was under 10 minutes.

[Jun 25, 2002, 12:50 PM] qaske Downtime

qaske(www250) crashed under high load and was rebooted. Downtime was under 10 minutes.

[Jun 25, 2002, 10:07 AM] kfin Downtime

kfin(www151) crashed due to high load and was rebooted. Downtime was under 10 minutes.

[Jun 25, 2002, 2:29 AM] ynilo Downtime

ynilo (www166) crashed after running out of swap space, and had to be rebooted. Total downtime was about 15 minutes.

[Jun 24, 2002, 11:52 PM] epsilon Downtime

epsilon (www3) became unresponsive and had to be rebooted. Total downtime was about 15 minutes.

[Jun 22, 2002, 9:19 PM] zelaza Downtime

zelaza (www247) crashed under heavy load and required a reboot. Downtime was 10 minutes.

[Jun 22, 2002, 9:10 PM] gessire Downtime

gessire (www273) crashed due to high load and was rebooted. Downtime was under 15 minutes.

[Jun 21, 2002, 4:57 PM] merau Downtime

merau (www245) failed under heavy load and required a reboot. Downtime was 10 minutes total.

[Jun 21, 2002, 1:32 PM] chi Downtime

chi (www14) crashed due to high load and was rebooted. Downtime was under 15 minutes.

[Jun 20, 2002, 8:21 PM] saghotta Downtime

saghotta (www259) was rebooted to clear an issue with the network interface card. Downtime was less than 5 minutes.

[Jun 20, 2002, 7:51 AM] paell Downtime

paell(www153) crashed due to high load and was rebooted. Downtime was under 5 minutes.

[Jun 19, 2002, 11:22 PM] enso Maintenance Completed

We have completed the drive swap on enso (www152). Total downtime for this maintenance was under 10 minutes.

[Jun 19, 2002, 4:41 PM] aqquz Downtime

aqquz(www248) crashed due to high load and was rebooted. Downtime was under 15 minutes.

[Jun 19, 2002, 2:35 PM] yiwen Downtime

yiwen (www246) crashed under a high load and was rebooted. The downtime was approximately 10 minutes.

[Jun 19, 2002, 2:07 PM] enso Planned Maintenance

We have observed a warning condition on the hard drive in enso (www152) which may indicate an imminent failure. Consequently, this evening after 6pm Eastern time, we will be taking the server offline to initiate a drive swap. There will be two brief periods of downtime, less than ten minutes each. Additional information will be posted once the swap is completed.

[Jun 19, 2002, 1:22 PM] merau Downtime

merau (www245) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Jun 18, 2002, 7:43 PM] ingwaz Downtime

ingwaz (www149) became unresponsive due to high load and was rebooted. Downtime was under 10 minutes.

[Jun 18, 2002, 5:12 PM] merau Downtime

merau(www245) crashed due to high load and was manually rebooted. Downtime was approximately 20 minutes.

[Jun 18, 2002, 7:26 AM] noldo Downtime

noldo (www56) crashed due to high load and was rebooted. Downtime was under 15 minutes.

[Jun 17, 2002, 1:16 PM] zelaza Downtime

zelaza crashed due to high load and was manually rebooted. Downtime was under 15 minutes.

[Jun 15, 2002, 5:46 AM] Network Maintenance Completed

The network maintenance of the Cable & Wireless circuit has been completed.

[Jun 15, 2002, 4:27 AM] Network Maintenance

Cable & Wireless is having an extended maintenance period, which is expected to be complete by 6AM. Our other providers are carrying the traffic as normal. We will post additional information when this maintenance is completed.

[Jun 14, 2002, 3:33 AM] ando Downtime

ando (www45) crashed under heavy load and had to be rebooted. Total downtime was under 10 minutes.

[Jun 12, 2002, 10:25 PM] emancholl Downtime

emancholl has again hung due to high load. We are closely monitoring this server and tracking the problem. Total downtime was under 10 minutes. We apologize for the inconvenience.

[Jun 12, 2002, 1:31 PM] emancholl Downtime

emancholl (www37) crashed under high load and had to be rebooted. Total downtime was under 10 minutes.

[Jun 11, 2002, 9:37 AM] yitiya Downtime

yitiya (www252) crashed under high load today. Downtime was under 10 minutes.

[Jun 10, 2002, 6:19 PM] zhun Maintenance Completed

We have complete the chassis swap of zhun (www107). It is now a Pentium 4 1.6GHz with 512Mb of RAM. Total downtime was under 10 minutes.

[Jun 10, 2002, 4:43 PM] zhun Planned Maintenance

We will be taking zhun (www107) down briefly tonight after 6PM for a chassis swap. Downtime is not expected to exceed 10 minutes. An additional notice will be posted once the maintenance is complete.

[Jun 10, 2002, 4:34 PM] foma Downtime

foma (www176) crashed and was rebooted. Total downtime was under 15 minutes.

[Jun 9, 2002, 4:37 PM] merau Downtime

merau (www245) froze under high load and had to be rebooted. Total downtime was under five minutes. We will continue to monitor this server closely.

[Jun 8, 2002, 10:29 PM]

tenshian (www103) crashed and had to be rebooted. The server will be closely monitored.

[Jun 7, 2002, 10:52 PM] ciot Downtime

ciot (www161) had to be rebooted due to a minor network problem. Total downtime was less than 5 minutes.

[Jun 7, 2002, 6:31 PM] qaske Downtime

qaske (www250) was rebooted to clear an issue with the network card. Downtime was under 5 minutes.

[Jun 7, 2002, 12:41 PM] Network Problems Resolved

At this time, all of our network circuits have returned to service. We are still awaiting a full explanation from Verizon as to the nature of the problem. We are continuing to monitor all circuits closely.

[Jun 7, 2002, 12:05 PM] Network Problems

Verizon has acknowledged a fiber cut in our area which has affected four of our circuits: Sprint, UUnet, Genuity, and Digex. They are currently working to correct the cut, but have not yet given an ETA. Our AT&T, Cable & Wireless, and Global Crossing circuits are on separate fiber and continue to operate. We will post more details as they become available.

[Jun 7, 2002, 11:31 AM] Network Problems

We are currently experiencing problems with several network circuits, possibly due to work by the local telephone company. We are working with all affected providers (Sprint, UUNet, Genuity, and Digex) to restore service as quickly as possible, and will post further details as they are available.

[Jun 5, 2002, 1:12 PM] quan Downtime

quan (www120) crashed under a high load and was rebooted. The downtime was under 15 minutes.

[Jun 4, 2002, 10:02 PM] emancholl Downtime

emancholl (www37) became unresponsive due to high load and was rebooted. Total downtime was less than 10 minutes.

[Jun 4, 2002, 8:28 PM] scire Downtime

scire (www78) became unresponsive due to high load and was rebooted. Downtime was less than 10 minutes.

[Jun 2, 2002, 8:58 PM] merau Downtime

merau (www245) crashed under high load and had to be rebooted. Total downtime was under 10 minutes.

[May 31, 2002, 6:22 PM] xoopi Maintenance Completed

Work has been completed on xoopi at this time. Total down time was less than 10 minutes.

[May 31, 2002, 3:38 PM] xoopi Emergency Maintenance

We have detected failing sectors on xoopi's hard drive, and are beginning a drive swap immediately to replace this drive before any problems occur. There will be two brief downtimes at the beginning and end of work. We shall post a second notice when work has been completed.

[May 30, 2002, 6:25 PM] DB7 Downtime

DB7 crashed and had to be rebooted. We are continuing to monitor the server.

[May 30, 2002, 4:30 AM] chi Downtime

chi (www14) crashed after running out of swap space, and had to be rebooted. Total downtime was about 10 minutes.

[May 28, 2002, 7:44 PM] yitiya Downtime

yitiya (www252) crashed under high load and had to be rebooted. Total downtime was less than 10 minutes.

[May 28, 2002, 7:27 PM] ACC Maintenance Update

At this time, all sections of the Account Control Center, with the exception of the Billing Interface, have been reopened. We are continuing work on the Billing Interface, and will post a notice when it has been reopened.

[May 28, 2002, 3:08 PM] ACC Maintenance

The Account Control Center has been taken offline for maintenance. We expect it to be back online within an hour. We apologize for any inconvenience.

[May 27, 2002, 8:21 PM] ciot Downtime

ciot (www161) was rebooted to clear a network interface problem. Downtime was less than 5 minutes.

[May 27, 2002, 11:57 AM] chi Downtime

chi(www14) crashed under high load. Downtime was under 15 minutes.

[May 27, 2002, 7:14 AM] quan Downtime

quan(www120) crashed under high load this morning. Downtime was under 15 minutes.

[May 26, 2002, 3:00 PM] chi Downtime

chi (www14) became unresponsive due to high load and was rebooted. Downtime was less than 10 minutes.

[May 25, 2002, 11:40 AM] Network Problems

Sprint has advised us that they are having an extended outage and maintenance window in their Maryland POP this morning. Although this has not affected our circuit specifically, several customers have reported difficulty in reaching our network through Sprint. Sprint expects to have the problem resolved by 12:30pm Eastern time.

[May 24, 2002, 4:04 PM] ando Maintenance Completed

The drive swap of ando has been completed.

[May 23, 2002, 9:42 PM] ando Planned Maintenance

ando will be taken down briefly this evening to begin a drive swap. There will be a second brief outage to finish the swap. Another notice will be posted when work has been completed.

[May 23, 2002, 9:14 AM] elhaz Downtime

elhaz (www144) crashed under heavy load, and had to be rebooted. Downtime was approximately 20 minutes.

[May 22, 2002, 10:12 AM] merau Downtime

merau (www245) was rebooted to clear up a high load condition. Downtime was approximately one minute.

[May 21, 2002, 10:16 PM] epsilon Downtime

epsilon (www3) became unresponsive due to high load and was rebooted. Downtime was less than 10 minutes.

[May 20, 2002, 4:17 PM] zelaza Downtime

zelaza (www247) was rebooted to clear up a network interface problem. The server will be monitored closely for any further problems. Downtime was under ten minutes.

[May 20, 2002, 1:38 PM] tiwaz Downtime

tiwaz (www146) crashed under heavy load, and had to be rebooted. Downtime was approximately 10 minutes.

[May 18, 2002, 12:08 AM] jera Maintenance Completed

The drive upgrade for jera is complete. It now has a 60GB drive. Total downtime for this upgrade was about 10 minutes.

[May 17, 2002, 5:31 PM] auma, falku Maintenance Completed

The drive upgrades on these servers are completed at this time.

[May 17, 2002, 1:10 AM] humph Maintenance Completed

The drive upgrade for humph has been completed, with new capacity at 40GB. total downtime was less than 10 minutes.

[May 16, 2002, 7:10 PM] mu Maintenance Completed

The drive swap on mu (www7) has been completed. It has a new 40GB drive. Total downtime for the drive swap was under 15 minutes.

[May 16, 2002, 5:41 PM] Server Planned Maintenance

The following servers will begin drive upgrades this evening:

• auma
• falku
• humph
• jera

Each server will be taken down for 5-10 minutes at the beginning and completion of work. Notices will be posted when work has been completed.

[May 16, 2002, 4:07 PM] ciot Maintenance Completed

We have completed the drive swap on ciot (www161). It now has a 40GB hard drive. Total downtime for this maintenance was about 10 minutes.

[May 16, 2002, 3:58 AM] shema Maintenance Completed

We have completed the drive swap of shema (www203). It now has a 40GB hard drive. Total downtime for this maintenance was under 10 minutes.

[May 15, 2002, 10:34 PM] emra Maintenance Completed

We have completed the drive swap on emra (www183). It now has a 40GB hard drive. Total downtime for this maintenance was under 10 minutes.

[May 15, 2002, 10:02 PM] ydhu Maintenance Completed

We have completed the drive swap on ydhu (www83). It now has a 40GB hard drive. Total downtime for this maintenance was under 10 minutes.

[May 15, 2002, 5:11 PM] epsilon Downtime

epsilon (www3) became unresponsive due to high load and was rebooted. Total downtime was around 5 minutes.

[May 15, 2002, 2:09 PM] Server Planned Maintenance

The following servers will begin drive upgrades this evening:

• emra
• ciot
• ydhu
• shema
• mu

Each server will be taken down for 5-10 minutes at the beginning and completion of work. Notices will be posted when work has been completed.

[May 13, 2002, 3:16 PM] merull Downtime

merull (www180) crashed under high load. Downtime was under 10 minutes.

[May 13, 2002, 12:41 AM] Network Problems Resolved

Our OC-3c to Sprint went down at 12:24am and came up at 12:36am. Most traffic shifted to alternate providers during this outage. We suspect problems in Sprint's network, but are pursuing further information from them directly.

[May 10, 2002, 8:11 PM] rho, flure, dhaame, kodh, fehu, gebo, bhoth Maintenance Completed

Work on these servers has been completed. fehu and gebo have additionally been upgraded to Athlon XP+ 1600 systems.

[May 10, 2002, 4:48 PM] Server Planned Maintenance

The following servers will be taken down briefly this evening for relocation in our datacenter:

• flure
• dhaame

Downtime should be brief, less than 10 minutes. Additionally, the following server will be taken down briefly for upgrades and maintenance:

• fehu

Downtime should again be less than 10 minutes.

[May 10, 2002, 2:13 PM] fehu Downtime

fehu (www134) crashed under heavy load and required a manual reboot. Down time was 15 minutes.

[May 10, 2002, 3:48 AM] kfin Planned Maintenance

The drive swap of kfin(www151) has been completed. The hardware was also upgraded at this time to an AMD Athlon XP1600, with 1024MB of RAM. Total downtime was about 15 minutes.

[May 10, 2002, 1:18 AM] glikk Maintenance Completed

We have completed the drive swap on glikk (www119). It now has a 40GB hard drive. We also took this opportunity to upgrade it's hardware to a Pentium III 1000MHz with 512MB of RAM. Total downtime for this maintenance was under 10 minutes.

[May 9, 2002, 11:25 PM] xi Maintenance Completed

We have completed the drive swap on xi (www8). It now has a 40GB hard drive. Total downtime for this maintenance was under 5 minutes.

[May 9, 2002, 7:25 PM] kirlian Downtime

Due to a failed power cable the server was rebooted. Downtime was about 10 Minutes.

[May 9, 2002, 4:27 PM] thurisaz Maintenance Completed

We have completed the drive swap on thurisaz (www136). It now has a 40GB hard drive. Total downtime for this maintenance was under 10 minutes.

[May 9, 2002, 3:28 PM] Server Planned Maintenance

Tonight, after 6PM EST, the following servers will be taken down briefly to begin drive upgrades:

• kfin
• bhoth
• gebo
• rho
• glikk
• xi
• kodh

Initial downtime should be less than 10 minutes. This maintenance is a part of on-going upgrades to increase server capacity and preformance. Additional notices will be posted when work has been completed.

[May 9, 2002, 3:24 PM] ansuz Maintenance Completed

We have completed the drive swap on ansuz (www137). It now has a 40GB hard drive. We also took this opportunity to upgrade it's hardware to a Pentium III 1000MHz with 512MB of RAM. Total downtime for this maintenance was under 10 minutes.

[May 9, 2002, 3:11 PM] uumor Maintenance Completed

We have completed the drive swap on uumor (www159). It now has a 40GB hard drive. Total downtime for this maintenance was under 5 minutes.

[May 8, 2002, 1:15 PM] thurisaz, ansuz, uumor Planned Maintenance

Tonight after 6PM EST, we will be taking the following servers down briefly to begin drive swaps:

thurisaz (www136)
ansuz (www137)
uumor (www159)

Initial domntime for each server should be no more than 10 minutes. This swap is to increase drive capacity of each server. Additional notices will be posted as more information becomes available.

[May 8, 2002, 4:03 AM] gamma Downtime

gamma(www2) became unresponsive and was rebooted. Total downtime was about 10 minutes.

[May 3, 2002, 2:24 PM] aaze Emergency Maintenance

aaze (www65) will be taken down briefly for a physical relocation. Expected downtime is less than 10 minutes.

[May 3, 2002, 12:41 PM] yitiya Downtime

yitiya (www252) crashed under heavy load, and was rebooted. Downtime was approximately 10 minutes.

[May 2, 2002, 5:08 PM] fehu Downtime

fehu (www134) crashed again under high load. Downtime was under 10 minutes. We are watching the server closely to determine the source of the problems.

[May 2, 2002, 4:10 PM] fehu Downtime

fehu (www134) crashed under high load and need to be rebooted manually. Downtime was under 15 minutes.

[May 2, 2002, 12:30 PM] yitiya Downtime

yitiya (www252) crashed under heavy load and had to be rebooted. Total downtime was under 10 minutes.

[May 1, 2002, 6:33 PM] neter Emergency Maintenance

Due to a problem with the ethernet interface, we have taken neter (www132) down briefly to swap out the faulty hardware. Total downtime for this emergency maintenance was under 10 minutes.

[May 1, 2002, 1:16 PM] Sprint Peering

Sprint has advised us that they are having some trouble reaching PSInet (which may now include Cogent) destinations through their network. They expect to have this problem, which is affecting a few customers, resolved shortly; if they do not do so, we may choose to shift that traffic to another provider for some period of time.

[Apr 29, 2002, 4:30 AM] hwesta Maintenance Completed

After some reconstruction work, hwesta (www51) is now up and running in a new chassis, with no loss of user data. Please accept our apologies for the unforeseen downtime.

[Apr 29, 2002, 3:27 AM] hwesta Emergency Maintenance

hwesta (www51) is refusing to boot in a new chassis. We are in the process of swapping in a new hard drive. We will post further details as they develop.

[Apr 29, 2002, 3:10 AM] hwesta Emergency Maintenance

The CPU fan on hwesta (www51) has failed, and the system is therefore refusing to boot. We are in the process of swapping the server into a new chassis, and expect to have it back online within the next 30 minutes.

[Apr 25, 2002, 8:19 PM] ampa Downtime

ampa (www52) was rebooted to clear up a swap issue. Total downtime was about 5 minutes.

[Apr 25, 2002, 12:47 PM] Network Problems

UUnet has posted a notice on their Web site indicating "Routing issues on the backbone". Traffic on the circuit is only slightly below normal at this time, so we will take no further action unless there are further outages or developments.

[Apr 25, 2002, 11:52 AM] Network Problems

Starting this morning at 10am Eastern time, there have been nationwide reports of slowdowns and outages on the UUnet backbone. At 10:10am and at 11:45am, we saw our UUnet traffic drop precipitously. This traffic is not being taken up by other networks because the traffic is sourced from UUnet's customers and is being lost due to their backbone difficulties.

There has been no official announcement from UUnet, and their network status page does not indicate that these outages have taken place or that they are working to correct them. This problem has been confirmed by a wide variety of postings to the NANOG mailing list.

If the problems persist today, we will consider turning down our UUnet OC-3c circuit in order to minimize the impact on UUnet peers we reach through their network. There is no similar solution for direct UUnet customers, however, unless they are also multi-homed.

If further information is forthcoming from UUnet, we will post it.

[Apr 24, 2002, 7:21 PM] ando Emergency Maintenance

Due to failing RAM, we have taken ando (www45) down briefly for a RAM swap. Total downtime for this emergency maintenance was under 5 minutes.

[Apr 24, 2002, 1:57 AM] ando Downtime

ando (www45) crashed under heavy load and had to be rebooted. Total downtime was about 20 minutes.

[Apr 23, 2002, 4:36 PM] csor Downtime

csor(www190) crashed under heavy load. Downtime was under 15 minutes.

[Apr 23, 2002, 2:37 PM] Server Maintenance

Beginning at 5pm Eastern time today, we will be rebooting all user servers in order to deploy several small security fixes. In general, most servers will reboot within five minutes; a few may take as long as fifteen minutes. We do not expect any longer disruption of service.

We apologize for the inconvenience, but keeping up-to-date on any security fixes which could impact our system availability is an important part of the service we provide to our customers.

[Apr 23, 2002, 2:33 PM] seyyav Downtime

seyyav (www238) crashed under high load this afternoon, downtime was under 10 minutes.

[Apr 23, 2002, 7:34 AM] yanta Downtime

Yanta crashed due to high sever loads. Approximate downtime was five minutes.

[Apr 22, 2002, 2:35 PM] yiwen Downtime

yiwen(www246) crashed under heavy load this afternoon, downtime was under 10 minutes.

[Apr 22, 2002, 12:17 PM] ando Downtime

ando (www45) crashed under high load, and required an extensive filesystem check. Downtime was approximately 20 minutes.

[Apr 19, 2002, 3:29 PM] nuin, neter Maintenance Completed

Upgrades on these servers have been completed at this time.

[Apr 18, 2002, 10:22 PM] nuin, neter Planned Maintenance

These servers will be taken down this evening briefly for hardware upgrades. There will be two such downtimes to complete this work. A second notice will be posted once this has been completed.

[Apr 18, 2002, 11:55 AM] Network Maintenance Completed

Our upgrade from DS-3 to OC-3c service with AT&T has gone smoothly, and we are now serving data over the new link. We have observed improved performance, as well as increased capacity, naturally.

If you are having any difficulty reaching our network, please contact us at urgent@pair.com.

[Apr 18, 2002, 10:38 AM] Network Maintenance

Beginning shortly after 11am Eastern time today, we will be upgrading our AT&T DS-3 service to OC-3c service. Traffic will be carried by our other providers during the switch, so no disruption is expected. This upgrade will significantly increase our overall capacity, and is part of our ongoing program to expand and improve our world-class network.

[Apr 16, 2002, 10:56 PM] lauta Downtime

lauta (www260) crashed under high load and had to be rebooted. Total downtime, approx. 20 minutes.

[Apr 15, 2002, 12:24 PM] kanat Downtime

kanat (www128) was rebooted to clear up a problematic resource condition. Downtime was approximately one minute.

[Apr 15, 2002, 10:12 AM] tisibo Downtime

tisibo(www265) crashed under heavy load. Downtime was under 10 minutes.

[Apr 14, 2002, 5:58 PM] aaze Downtime

Server aaze (www65) was rebooted this evening, and has since returned to normal operation. Total downtime was less than 5 minutes.

[Apr 11, 2002, 7:03 PM] ampa Downtime

ampa(www52) crashed under high load and was manually rebooted. Downtime was under 10 minutes.

[Apr 11, 2002, 11:04 AM] enek Downtime

enek (www130) was rebooted to clear up a problematic load condition. Downtime was approximately one minute.

[Apr 10, 2002, 4:28 PM] quan Maintenance Completed

The emergency maintenance on quan (www120) has been completed. No user data was lost.

[Apr 10, 2002, 10:19 AM] quan Maintenance Update

The restoration of files on quan (www120) is continuing at this time. The server has been brought back online, and files are being served by the old drive until they are copied. We will post another notice when the maintenance is complete.

[Apr 10, 2002, 9:42 AM] pyyl Downtime

pyyl (www97) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Apr 10, 2002, 9:19 AM] quan Maintenance Update

We have been able to access the failing hard drive on quan (www120) and are currently restoring the files to a new drive. We expect the restoration to be completed within an hour.

[Apr 10, 2002, 8:49 AM] quan Emergency Maintenance

quan (www120) has suffered a hard drive failure, and we are working to restore service as quickly as possible. More information will be posted as it is available.

[Apr 10, 2002, 7:37 AM] thurisaz Downtime

thurisaz crashed under high load, and had to be manually rebooted. Downtime was approximately 10 minutes.

[Apr 9, 2002, 9:52 PM] omega Downtime

omega (www16) rebooted this evening and was down for 10 minutes. It has since returned to normal operation.

[Apr 9, 2002, 11:32 AM] emancholl Downtime

emancholl (www37) crashed under high load, and had to be manually rebooted. Downtime was approximately 10 minutes.

[Apr 8, 2002, 3:03 PM] emancholl Downtime

emancholl(www37) had to be rebooted this afternoon. Downtime was under 15 minutes.

[Apr 8, 2002, 2:42 PM] uinus Downtime

uinus(www228) had to be rebooted this afternoon. Downtime was under 15 minutes.

[Apr 8, 2002, 12:22 PM] qaske Downtime

qaske(www250) was rebooted this morning. Downtime was under 15 minutes.

[Apr 6, 2002, 5:53 AM] ciot Downtime

ciot (www161) crashed under heavy load and had to be rebooted. Downtime was about 20 minutes.

[Apr 4, 2002, 3:36 PM] enda Downtime

enda (www80) became unreachable under heavy load and was rebooted. Total downtime was 10 minutes.

[Apr 3, 2002, 2:59 PM] vepar Downtime

vepar (www201) crashed under high load. Downtime was approximately 10 minutes.

[Apr 2, 2002, 11:35 AM] athani Downtime

athani(www157) was rebooted this morning, downtime was under 10 minutes.

[Apr 2, 2002, 11:31 AM] agho Downtime

agho.pair.com (www187) was rebooted to clear up a high load condition. Downtime was approximately one minute.

[Mar 29, 2002, 4:22 PM] Network Problems Resolved

Our Savvis line has returned to full service again at this time. We are still awaiting a full report on exactly what the problem was. We will post a followup once we have this information.

[Mar 29, 2002, 3:14 PM] Network Problems

SAVVIS is still working to identify the cause of the outage of their DS-3 circuit, which began around 11:30am Eastern time today. Although the circuit appears to be up, it refuses to pass traffic in either direction. Telco employees have been dispatched to verify the cross-connect in a nearby local facility.

This circuit downtime has not affected our overall connectivity, nor would it do so even under peak usage. Customers should see no performance problems as a result of this outage. When the situation is resolved, we will post further information.

[Mar 29, 2002, 12:57 PM] Network Problems

At 11:40 am EST, we lost contact with our Savvis DS-3. All traffic should be transparantly diverted to our other lines, offering no loss of service. We are awaiting to hear from Savvis with a resolution of this issue.

[Mar 29, 2002, 12:47 PM] yitiya Maintenance Completed

We have completed the drive swap on yitiya (www252). Total downtime was about 40 minutes.

[Mar 29, 2002, 7:13 AM] delsi Downtime

delsi (www206) had to be manually rebooted this morning. Downtime was under 15 minutes.

[Mar 28, 2002, 10:55 AM] yitiya Planned Maintenance

We have detected errors on the hard drive in yitiya.pair.com (www252). We will be starting a hard drive swap at approximately 3:00PM Eastern. There will be two periods of downtime, neither of which should exceed 10 minutes. We will post more information as it becomes available.

[Mar 28, 2002, 7:33 AM] delsi Downtime

Due to high load. Approximate down time was 25 minutes.

[Mar 28, 2002, 2:05 AM] zhun Downtime

zhun (www107) crashed under heavy load and had to be restarted. Downtime was about 10 minutes.

[Mar 23, 2002, 2:59 PM] db26 Restored to Service

After a quick drive swap, db26.pair.com has been restored to service. All user data was transferred from the dying hard drive so there should be no loss, as we did not have to go to the four-hour-old backups. If you encounter any problems working with db26, please e-mail us.

[Mar 23, 2002, 1:59 PM] db26 Emergency Maintenance

db26 has had a catastrophic drive failure. We are working on restoring services for db26 as soon as possible. Expected time to completion is 2 hours.

[Mar 21, 2002, 11:36 PM] khurla, paell Maintenance Completed

The upgrades for these servers have been completed.

[Mar 21, 2002, 8:09 PM] tehun Downtime

tehun(www226) had to be rebooted due to high load on the server. Downtime was less than 10 minutes.

[Mar 20, 2002, 1:50 PM] analog Security Update

This morning we have upgraded analog5 on all of our servers to the newest revision, version 5.22. This action was taken in response to a security advisory posted by the author of the utility. Although there was no danger to the server itself, an attacker could potentially inject javascript code into analog's output in a "cross-site scripting" attack. If you are using the Account Control Center to create your reports, you do not need to take any action.

[Mar 19, 2002, 11:00 PM] straif, cele Maintenance Completed

The drive upgrades for these servers have been completed.

[Mar 19, 2002, 9:20 PM] bemnet Planned Maintenance

Due to recent instability, bemnet has been swapped to a new server chasis. We will continue to monitor this server in case of problems.

[Mar 19, 2002, 2:54 PM] emancholl Downtime

emancholl (www37) was rebooted to clear up a high load condition. Downtime was approximately one minute.

[Mar 18, 2002, 11:23 PM] Server Upgrades

The following servers will be taking down briefly this evening to begin procedure for drive upgrades:

• cele
• khurla
• paell
• straif

There will be two brief periods of downtime, at the beginning and end of work. Another notice will be posted when work has been completed.

[Mar 18, 2002, 5:03 PM] yitiya Downtime

yitiya (www252) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Mar 18, 2002, 2:05 PM] SSH Service Updated

In order to resolve minor incompatibilities, we have deployed the most current stable version of the OpenSSH daemon. SSH service should not be disrupted by this upgrade. As always, if you encounter any differences in behavior as a result of this upgrade, please inform us at urgent@pair.com.

[Mar 18, 2002, 12:47 PM] bemnet Downtime

bemnet (www158) crashed under a high load and was rebooted. The downtime was under 15 minutes.

[Mar 15, 2002, 12:57 PM] ungwe Downtime

ungwe (www48) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Mar 15, 2002, 9:05 AM] auddo Downtime

auddo(www253) was rebooted to correct a memory allocation error. Downtime was under 5 minutes.

[Mar 15, 2002, 8:04 AM] emancholl Downtime

emancholl(www37) crashed under high load and was rebooted from the console. Downtime was under 10 minutes.

[Mar 14, 2002, 2:45 PM] bemnet Downtime

bemnet (www158) became unresponsive, and was restarted. After extensive filesystem checks, it has returned to service.

[Mar 13, 2002, 3:40 PM] yitiya Downtime

yitiya (www252) crashed under a high load and was rebooted. The downtime was under 20 minutes.

[Mar 11, 2002, 3:24 PM] foma Downtime

foma (www176) crashed under high load this afternoon and rebooted itself. Total downtime was under 10 minutes.

[Mar 8, 2002, 10:00 AM] sshd Security Update

A patched sshd (secure shell daemon) was deployed to all user servers yesterday after a security vulnerability in OpenSSH was made public. If you are experiencing any problems connecting via SSH, please let us know at support@pair.com. We've received a report of a problem with using RSA keys and SSH version 2, which we are currently investigating.

[Mar 7, 2002, 3:14 PM] bemnet Downtime

bemnet (www158) crashed under a high load and was rebooted. The downtime was 15 minutes.

[Mar 5, 2002, 5:42 PM] yitiya Downtime

yitiya (www252) was rebooted to clear up a problematic high load condition. Downtime was approximately 2 minutes.

[Mar 5, 2002, 12:01 PM] bemnet Downtime

bemnet(www158) crashed under heavy load and had to be rebooted from the console. Total downtime was under 20 minutes.

[Mar 2, 2002, 7:21 PM] merau Downtime

merau (www245) was rebooted to clear out minor network problems. Total downtime was less than 10 minutes.

[Mar 2, 2002, 12:04 PM] epsilon Downtime

epsilon (www3) was unresponsive due to high load. It was rebooted, and had to perform a filesystem check. Total downtime was under 15 minutes.

[Mar 1, 2002, 11:48 AM] Update on Apache and PHP Updates

For the most current information on our Apache/PHP update, please visit: http://www.pair.com/pair/support/notices/php412.html

We are working to resolve the outstanding issues documented there.

[Mar 1, 2002, 8:56 AM] Apache/PHP Update Complete

As of 9am Eastern time, the upgrade to Apache and PHP on all shared servers has been completed. The file upload functionality has been restored, with no known security vulnerability remaining. The new build is Apache 1.3.23 with PHP 4.1.2 and Zend Optimizer 1.2.0.

Testing has revealed no significant changes in functionality. However, in our experience, any such major upgrade will always "break" something. We are ready to assist any customer encountering such issues. Please contact us at urgent@pair.com if you are experiencing new difficulties related to PHP or Apache.

We appreciate our customers' collective patience during this difficult upgrade. Normally, we prefer to make such upgrades on a more carefully considered schedule, although we understand that in this case the upgrade could not come quickly enough.

[Feb 28, 2002, 4:35 PM] Update on PHP Functionality

We have completed new builds of Apache 1.3.23 with PHP 4.1.2 for regular Web service, SSL-enabled Web service, and the standalone php4.cgi binary. Testing revealed several issues at the last minute, and consequently further work was required.

As a rule, we are not comfortable deploying critical updates late in the day. The greatest possible support resources will be available on Friday morning; consequently, we plan to deploy these code updates on Friday morning at approximately 8am Eastern time. Although we do not expect any functionality to break as a result of the update, these are significant updates that go beyond merely patching the recent security vulnerability, and it is always likely that there will be some impact.

We will be putting the code through further testing overnight, as an added precaution. We appreciate your patience in this difficult situation.

[Feb 28, 2002, 3:15 PM] foma Downtime

foma (www176) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Feb 28, 2002, 2:25 PM] PHP 3.0.18 Deployment

We have deployed a new build of PHP 3.0.18 to all user servers. This will protect PHP3 users from the recent security vulnerability. We do not believe there has been any other change in functionality for PHP3 with this new release.

We have developed a build of Apache 1.3.23 with PHP 4.1.2, which has the security vulnerability fixed, and are currently testing this. We are planning to deploy this new code within the next hour, which will allow us to restore the file upload capability for PHP4.

We appreciate your patience during these updates.

[Feb 27, 2002, 10:11 PM] Emergency Security Update

Within the last 24 hours, details of a vulnerability in PHP which can be exploited remotely have been made public. The vulnerability allows any attacker to send a malformed POST request to a PHP-enabled Web server in a manner that will allow remote access as the user running the Web server processes. In the general case on our servers, this means the "nobody" user.

Although the "nobody" user has limited privileges, any such access is a potential launching point for other nefarious activities. Moreover, some customers may be using a PHP with cgiwrap, meaning that their actual account is vulnerable because of this weakness.

We are working to deploy and test a new build of Apache that will include PHP 4.1.2, the version created specifically to address this vulnerability. However, this requires careful testing and can not be deployed immediately. In the interim, therefore, we have disabled the file upload feature of PHP on our servers. This is the quick workaround recommended by PHP developers and the CERT advisory. We are also contacting all customers who are using custom PHP builds, and recommending that they take similar steps until such time as they can deploy PHP 4.1.2.

We understand that this change interferes with functionality for some customer sites. We will have the new Apache+PHP build in place as soon as possible, and will post a further notice at that time. We ask that our customers respect our insistence on treating security vulnerabilities as problems no less critical than system outages.

For more information, please visit:
http://www.cert.org/advisories/CA-2002-05.html
http://security.e-matters.de/advisories/012002.html

We appreciate your patience and understanding.

[Feb 27, 2002, 4:08 PM] ungwe Downtime

A user script was causing frequent crashes on ungwe. The script has been identified and disabled. This caused intermittant downtime for approximately 25 minutes, while we tracked which script was causing the problems.

[Feb 25, 2002, 3:07 PM] mu Downtime

mu (www7) crashed under a high load and was rebooted. The downtime was approximately 15 minutes.

[Feb 25, 2002, 2:55 PM] bemnet Downtime

bemnet (www158) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Feb 25, 2002, 12:26 PM] epsilon Maintenance Completed

The hard drive swap for epsilon (www3) has been completed. Total downtime was approximately 10 minutes.

[Feb 24, 2002, 9:20 PM] bemnet Downtime

bemnet (www158) crashed under load and had to be restarted. Total downtime was about 15 minutes.

[Feb 24, 2002, 6:10 PM] epsilon Emergency Maintenance

We have observed a warning condition on the hard drive in epsilon (www3) which may indicate an imminent failure. We will be taking this server offline briefly to initiate a drive swap. There will be two brief periods of downtime, less than ten minutes each. Further information will be posted as it becomes available.

[Feb 23, 2002, 12:40 AM] bemnet Downtime

bemnet (www158) crashed under load and had to be restarted. Total downtime was about 15 minutes.

[Feb 19, 2002, 2:51 PM] cicka Maintenance Completed

The maintenance on cicka.pair.com (www167) was completed in the expected time frame. After an extensive filesystem check, the server appears to be performing correctly. We will continue to closely monitor this server to make certain everything is working as expected.

[Feb 19, 2002, 2:11 PM] cicka Emergency Maintenance

Due to problems which indicate a pending failure, we will be rebooting cicka.pair.com (www167) to perform filesystem tests. Downtime should be under 10 minutes. We will post more information as it becomes available.

[Feb 18, 2002, 11:02 PM] Network Problems Resolved

As of 10:45pm Eastern time, our AT&T and Global Crossing DS-3 circuits have been restored to service. Both circuits are carried by AT&T Local Services (formerly TCG), and during circuit grooming this evening, the cross-connects for these circuits were inadvertently removed. The problem was tracked down by both companies, and corrected by the AT&T Local Services engineers.

During the outage, customer traffic was carried by our other providers; we maintain significant excess capacity specifically in order to accommodate potential problems such as this. All service has now returned to normal.

[Feb 18, 2002, 9:21 PM] Network Problems

At 8:19pm Eastern this evening, our AT&T and Global Crossing DS-3 circuits went out of service. We believe the cause is an AT&T fiber cut in the Pittsburgh area. We have tickets open with both providers, and are urging a speedy resolution.

We have sufficient excess capacity to be unaffected by this outage, even if it runs for an extended period. We do expect the circuits to be back in service tonight, however.

[Feb 18, 2002, 12:00 PM] mhoh Downtime

mhoh (www191) crashed under a high load and was rebooted. The downtime was 10 minutes.

[Feb 16, 2002, 12:10 PM] epsilon Maintenance Completed

The drive upgrade to epsilon has been completed.

[Feb 15, 2002, 6:03 PM] Maintenance Completed

The drive upgrades on enso, elhaz, and vepar have been completed. Enso has additionally been upgraded to an Athlon 1000.

[Feb 15, 2002, 12:21 PM] mhoh Downtime

mhoh (www191) crashed under a high load and was rebooted. The downtime was approxmiately 10 minutes.

[Feb 14, 2002, 5:19 PM] Planned Maintenance

This evening, the following servers will be taken down briefly to begin the process of a drive upgrade:

• epsilon
• elhaz
• enso
• vepar>

These upgrades will allow for greater capacity, as well as the proactive replacement of older hardware. There will be two short downtimes for each server, as work is done. A second notice will be posted when work has been completed.

[Feb 13, 2002, 12:10 PM] mhoh Downtime

mhoh (www191) crashed under a high load and was rebooted. The downtime was approxmiately 10 minutes.

[Feb 12, 2002, 1:38 PM] mhoh Downtime

mhoh (www191) crashed under a high load and was rebooted. The downtime was under 15 minutes.

[Feb 11, 2002, 3:02 PM] csor Downtime

Rebooted due to high load. Downtime was less than 10 minutes.

[Feb 11, 2002, 10:14 AM] bemnet Downtime

bemnet (www158) crashed under heavy load and required a systems check before rebooting. Downtime was about 20 minutes. We will be keeping a close eye on the server for any unusual behavior.

[Feb 9, 2002, 1:33 PM] epsilon Downtime

epsilon (www3) crashed under a high load, and required an extensive filesystem check before coming back online. The total downtime was about 17 minutes.

[Feb 8, 2002, 4:45 PM] mini Downtime and Emergency Maintenance

While investigating recent downtime on mini (www126), the server became unresponsive and had to be rebooted. We have been monitoring this server closely and believe that a memory swap may improve its stability. We will be taking the server down shortly to swap out its present RAM. Downtime is expected to be less than 5 minutes.

[Feb 7, 2002, 11:52 AM] isris Downtime

isris (www194) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Feb 7, 2002, 5:22 AM] ynilo Downtime

ynilo(www166) crashed after running out of swap space and had to be restarted. Total downtime was about 10 minutes.

[Feb 6, 2002, 10:52 AM] iruvul Downtime

iruvul (www239) crashed under a high load and was rebooted. The downtime was under 15 minutes.

[Feb 5, 2002, 8:41 PM] bemnet Downtime

bemnet (www158) crashed under heavy load and required a manual disk check before coming back online. Downtime was 20 minutes.

[Feb 5, 2002, 2:55 PM] fehu Downtime

fehu (www134) crashed under heavy load. Downtime was less than 10 minutes.

[Feb 5, 2002, 12:57 AM] mini Downtime

mini(www126) crashed under heavy load and had to be rebooted. Total downtime was about 10 minutes.

[Feb 3, 2002, 3:55 PM] siyazi Downtime

siyazi (www251) was rebooted to clear up a network interface problem. We will be monitoring the server closely for any further problems. Downtime was less than five minutes.

[Feb 2, 2002, 10:28 AM] Network Maintenance

On Sunday afternoon, between 4pm and 7pm Eastern time, we will be moving our UUnet OC-3c circuit from legacy Cisco equipment to one of our Juniper gateway routers. Downtime for the circuit should not exceed 15 minutes, during which time other providers will easily carry our traffic.

This is an upgrade which will improve performance and reliability, while increasing potential future usage of this circuit. We are continuing to expand our network to satisfy customer demand.

[Feb 1, 2002, 1:17 PM] uumor Configuration Problem

A configuration error on uumor (www159) resulted in an interruption in mail and web service for domains on that server. The problem lasted for approximately 60 minutes. We have located and fixed the source of the error, and it should not recur. We apologize for the inconvenience.

[Jan 26, 2002, 4:24 AM] ceirt Maintenance Completed

The restoriation of ceirt (www39) has now been completed. Again, we apologize for the inconvenience this downtime has caused, and appreciate your patience. We also upgraded the server during this emergency maintenance. It is now a Pentium III 1000MHz, with 256MB of RAM, and a 60GB hard drive.

[Jan 26, 2002, 3:03 AM] ceirt Update

The server restoriation of ceirt.pair.com (www39) is still ongoing. Additional information will be posted as it becomes available. We apologize for the inconvenience this downtime has caused, and appreciate patience as we work to restore this server.

[Jan 25, 2002, 9:46 PM] ceirt Emergency Maintenance

Shortly after the original swap of ceirt's failing hard drive began, the original drive failed catastrophically. We are currently in the process of rebuilding ceirt from backups. This process should be complete within the next two hours. Access to ceirt will be restricted until that time. We will post further information as it becomes available.

[Jan 25, 2002, 7:41 PM] ceirt Emergency Maintenance

ceirt (www39) will be down for the duration of the emergency maintenance due to the nature of the errors. At this time we suspect that the downtime will be a few hours.

[Jan 25, 2002, 7:18 PM] ceirt Emergency Maintenance

We have observed a warning condition on the hard drive in ceirt (www39) which may indicate an imminent failure. We will be taking it down briefly to initiate an emergency drive swap. There will be two brief periods of downtime, less than ten minutes each. Additional information will be posted as it become available.

[Jan 25, 2002, 10:35 AM] db7.pair.com Downtime

db7.pair.com crashed this morning around 8:30am. It was rebooted manually and downtime was under 10 minutes.

[Jan 25, 2002, 12:55 AM] Server Upgrades Completed

The upgrades for this evening have been completed as follows:

• auma - add RAM to 512MB
• orvni - add RAM to 512MB
• or - Processor P-III 1000, Ram 512MB

[Jan 24, 2002, 11:08 PM] Server Reboots

This evening, all user servers were rebooted to install a system kernel upgrade. This update includes recent security patches and updates. All servers have returned to normal service, and downtime for each was less than 5 minutes.

[Jan 24, 2002, 4:03 PM] Server Upgrades

This evening, the following servers will be taken down briefly for hardware upgrades to improve preformance and capacity:

• auma
• or

A notice will be posted again once work has been completed.

[Jan 23, 2002, 11:01 AM] qaske Downtime

qaske (www250) crashed under a high load and was rebooted. The downtime was approximately 10 minutes.

[Jan 21, 2002, 10:36 PM] Security Reminder

We would like to remind all customers that we will never ask you for login information for any of your accounts or domains. As system administrators, there is no legitimate reason for anyone working here to ever ask for that information. Many online providers are occasionally targeted by scammers who, posing as administrators, will claim that they need this information for whatever reason.

If you receive any such notice, please forward it with full headers to abuse@pair.com immediately. We are investigating recent incidents and taking all appropriate actions to respond to the problem. We appreciate that our customers understand that this is a scam, and that any official e-mail you receive from us regarding anything will be certified by matching information posted on our Web site.

[Jan 18, 2002, 11:47 AM] foma Downtime

foma (www176) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Jan 17, 2002, 11:32 PM] Upgrades Completed

The scheduled upgrades of 5 servers for this evening have been completed. Each has been upgraded to 512MB of RAM. The downtime for each was less than 5 minutes.

[Jan 17, 2002, 6:45 PM] sether Downtime

sether (www95) crashed due to low swap space. It has been rebooted. Downtime was 10 minutes.

[Jan 17, 2002, 3:00 PM] Server Upgrades

This evening, the following servers will be taken down briefly for hardware upgrades:

• elhaz
• mhoh
• iota
• vepar
• zelaza

Downtime should be minimal for each. These upgrades are to increase server capacity under high loads. Another notice will be posted when this work is completed.

[Jan 17, 2002, 1:36 PM] siyazi Downtime

siyazi (www251) crashed and was rebooted. The downtime was approximately 15 minutes.

[Jan 17, 2002, 1:33 AM] ynilo Downtime

ynilo(www166) crashed under heavy load and had to be rebooted. Total downtime was about 15 minutes.

[Jan 16, 2002, 9:56 PM] zhun Downtime

zhun (www107) crashed under high load and had to be manually FSCK'd and rebooted.

[Jan 16, 2002, 1:36 PM] csor Downtime

csor (www190) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Jan 16, 2002, 10:54 AM] xomi Downtime

xomi (www169) crashed under a high load and was rebooted. The downtime was under 10 minutes.

[Jan 15, 2002, 6:56 PM] xomi Downtime

xomi (www169) crashed under heavy load and required a manual file check before coming back online. Downtime was 20 minutes.

[Jan 15, 2002, 4:02 PM] londaste Downtime

londaste(www177) crashed and had to be manually rebooted. Downtime was under 10 minutes.

[Jan 15, 2002, 1:07 PM] ungwe Downtime

ungwe(www48) crashed due to high load and was manually rebooted. Downtime was under 10 minutes.

[Jan 12, 2002, 1:46 PM] Network Problems

After going more than a week with no further misbehavior from our Foundry switches, we suffered a major outage today at precisely 1pm Eastern time. During this outage, our switches disagreed about which switch should be routing which traffic, and consequently, most traffic went nowhere.

As instructed by Foundry engineering support previously, we captured extensive debugging information during the outage, and reloaded the suspected offending switch eighteen minutes after the outage began. Traffic returned to normal immediately, and we have chosen to leave the offending switch in service until the fault, hardware or software, can be identified and remedied. We are hopeful that this will happen promptly; Foundry support is involved at some of their highest levels.

We apologize for the inconvenience of this downtime and its extension during debugging. We are working to ensure that the best possible network is in place, both in physical and logical redundancy. Unfortunately, even redundancy protocols have their own failure modes.

We will also be adding an additional spare switch to our network so that the suspected offender can be rotated out as another form of long-term testing.

[Jan 10, 2002, 2:59 PM] iski Maintenance

In order to help with recent higher loads on this server, the system RAM has been increased to 512MB. This should help to avoid the recent crashes we have seen with iski.

[Jan 10, 2002, 2:23 PM] iski Downtime

iski (www242) crashed under a heavy load and was rebooted. The downtime was approximately 15 minutes. We apologize for any inconvenience this situation may cause and we appreciate your patience as we work toward a solution.

[Jan 8, 2002, 2:44 PM] iski Downtime

iski(www242) is experiencing difficulties again today. We are working to correct the situation as fast as possible. We apologize for any inconvenience this situation may cause and we appreciate your patience as we work toward a solution.

[Jan 8, 2002, 2:05 PM] eahta Downtime

eahta(www227) crashed under heavy load and was rebooted this afternoon. Downtime was under 10 minutes.

[Jan 7, 2002, 7:44 PM] iski Downtime

iski (www242) has been experiencing some difficulties today, and has crashed a number of times, requiring that it be restarted. We are currently in the process of tracking the cause of these interuptions in service, and will post more news as we learn more.

[Jan 7, 2002, 4:22 PM] iski Downtime

iski (www242) crashed under high load and was rebooted. Downtime was less than 10 minutes.

[Jan 7, 2002, 12:39 PM] xebte Downtime

xebte (www230) crashed due to high load this morning, dowtime was less then 15 minutes.

[Jan 6, 2002, 11:10 AM] aedde Downtime

aedde (www84) was rebooted after it become unresponsive. Because it was not rebooted cleanly, the server had to perform a filesystem check. Total downtime was under 15 minutes.

[Jan 4, 2002, 3:13 PM] iski Downtime

iski (www242) was rebooted after it became unresponsive. It has since returned to normal service.

[Jan 4, 2002, 9:33 AM] khin Downtime

khin (www184) had to be manually rebooted this morning, downtime was under 10 minutes.

[Jan 3, 2002, 3:35 AM] ceirt Downtime

ceirt(www39) crashed and was restarted after running self diagnostics. Downtime was about 20 minutes.

[Jan 3, 2002, 12:51 AM] delsi Downtime

Delsi (www206) crashed under a heavy load and was rebooted. The downtime was approximately 15 minutes.

[Jan 2, 2002, 1:53 PM] iski Downtime

iski (www242) was rebooted to clear up a problematic high load condition. Downtime was approximately 1 minute.

[Jan 2, 2002, 12:59 AM] ulwar Downtime

ulwar (www82) crashed under a heavy load and was rebooted. The downtime was under 10 minutes.

View Notices for 2001