This is the continuation of a story begun in “
The Move” in which our heroes
battle nefarious network gremlins in order to save
zoomshare from imploding under
its own weight.
By 7am, when everyone else was waking, I was
getting into bed. Other than a need for a good long
nap, all seemed well. The move from colocation to
colocation took a little longer
than planned, but
judging by the performance we saw before leaving
the building zoomshare was running better than ever.
As I closed in on being awake for 24 hours I knew I
didn’t have a taxing day ahead of me, I had planned
my schedule accordingly at least. However, I still
had some work to do and tops on that list was a
check-in with zoomshare in a few hours, so I set my
alarm for 9 am and closed my eyes.
I don’t remember the alarm going off.
10 am. I could have used some more sleep, but that
could wait just a few minutes. A quick check-in on
how zoomshare was handling the morning traffic and
then a few more hours of shuteye.
I don’t think I even got in my chair, let alone
logged into my computer.
I had voicemail. In fact I had a voicemail from
kree10 that was just a few
minutes old.
Not good news.
Not good news at all, in fact. He was on his way
back to the colocation facility. No one from the
office was able to connect to any zoomshare site,
verifying in turn that everything was in working
order. Moreover, it was looking like a good
percentage of our users were having issues as well,
which meant it wasn’t localized to just one network
connection or path.
I gave kree10 a call, He asked me to meet him and
to bring any spare any
network switches I might have
with me. He was bring one as well, just I case. So
much for getting more shuteye.
I dressed, I disassembled part of my home network,
unplugged my network switch, picked up my laptop
and headed out the door.
The Odd Couple
Why bring along a network switch? I believe by the
time I was on the road back from which I had come
peenworm had determined that
about 60% of the
network packets from our office
were being dropped on their way to the zoomshare
servers in their new home. Most of the equipment
was the same equipment that had been humming along
at the old location just a day before. Most, but
not all.
Some of the new equipment already in place before
the move included a new network switch for managing
network traffic between the various servers. When
all is working correctly network switches properly
inspect network data packets as they are received,
determine their source and destination and forward
the traffic appropriately. When things go wrong,
well…things sure didn’t seem to be going right.
The funny thing was peenworm had seen something
like this within the past few months at the old
facility as well. It wasn’t caused by new equipment
so much as by an uptake in network traffic. It was
a network communication issue, a
duplex mismatch to be exact.
Ok, see a network connection can be unidirectional
or bidirectional. In either case this is known as
duplexing. Our networking equipment, our switch
helps determine if there is either a bidirectional
"two-way path" between the two connected parties or
a unidirectional "reverse path”. Just as it manages
the who, what and where of the network traffic the
switch helps manage the how. But if any of the
equipment on the network is misconfigured, then
boom, a network collision and network data
disappears, literately into the
ether.
In any case the quickest way to straighten out the
issue was going to be swapping out the new network
switch with another model. Traffic on the network
would right itself, packets would stop getting lost
and after a little rest we could property
reconfigured the newer network switch and place it
back online when ready.
If only.
After meeting up with kree10 we tried our switches.
One network switch replacement later and no
improvement. Two networks switches later and still,
no improvement. Something else was causing the
network loss. But what?
I Dare Say Mr. Holmes
In "
The Move" I mentioned feeling
out of place as a software engineer in a network
engineer’s world. In the world of computing there
are hardware and software "layers" that allow
engineers to develop ever more powerful tools. Each
"layer of abstraction" removes a level of
complexity out of one’s hands such that other
problems can be tackled. As a web developer, for
example, on any given day I don’t have to worry
about programming networking protocols to mange
duplexing issues, as that’s all been taking care
for me by someone else.
A third wheel I said I felt like, sort of like Dr.
Watson tagging along as Sherlock Holmes probes his
client’s inner thoughts in search for clues … and
an answer.
And yet in Arthur Conan Doyle’s tales Dr. Watson
was a medical man, a practitioner of science. He
too could put his analytical problem solving skills
to the test, even in unfamiliar waters. After all
Holmes’ axiom rings true in just about any logical
situation, “Once you eliminate the impossible,
whatever remains, no matter how improbable, must be
the truth.”
If its not the network switch it must be some other
piece of equipment. Alas, logic has a way of
escaping one’s sense after 36 hours and just a few
hours of sleep. Frustration starts to set in, and
that sure was the case for all three of us after a
few more hours of troubleshooting. The office was
having issues connecting, but at the facility where
we plugged into the net everything worked better
than fine.
We had tried everything we could. It sure seemed
like it wasn’t a problem with any of our equipment.
Therefore, as Mr. Holmes would correctly point out,
it must be a problem elsewhere. When the floodgates
were opened and all of the traffic trying to get to
zoomshare came rushing in, something starting
acting up.
But we couldn’t think straight; perhaps our logic
was escaping us. We must have missed something. It
couldn’t be a problem with the colocation facility
could it? We didn’t see any network issues.
Could it be with our new Internet service provider?
The tech we talked to sure didn’t see any issues
from his workstation.
Any yet ... To be continued in “
The Analogy”
