ATTN builder admins: all x86 builders down

[Jarod Wilson]

On Wed, Jan 20, 2010 at 6:33 AM, Adrian Reber <adrian at lisas.de> wrote:
> On Tue, Jan 19, 2010 at 10:13:01PM -0500, Jarod Wilson wrote:
>> >>>>> As of Tue Jan 19 23:27:20 UTC 2010 all x86 builders seem to be down,
>> >>>>> so it's impossible to run any builds for rawhide. Please fix ASAP.
>> >>>>
>> >>>> Unfortunately right.
>> >>>> Builders are unreachable from  our rpmfusion.org
>> >>>> I'm still looking at builders owner to get more info about what the
>> >>>> hell is going on, sorry for the trouble.
>> >>>
>> >>> Mine should register itself again in just a sec. I had a double-disk
>> >>> failure in the machine that used to host the builder vm, so its been
>> >>> transplanted, but I didn't get dns and/or iptables forwarding on the
>> >>> host quite right, apparently.
>> >>
>> >> Bah. No, that's not it either, plauge-builder service keeps dying for
>> >> some reason:
>> >>
>> >> # service plague-builder status
>> >> plague-builder dead but pid file exists
>> >>
>> >> Investigating...
>> >
>> > Was failing to start due to /var/lib/mock/plague-builder not existing.
>> > No clue why it wasn't there, but after creating it, things started up
>> > just fine. However, its still not registering yet. Dunno if I should
>> > just give it a bit or what... Need to get my kids in bed, will do that
>> > real quick, then check back in.
>>
>> Okay, I got nothin'. Still not registering, apparently, despite my
>> best efforts. I've verified that port 8888 and 8889 are indeed
>> reachable from the outside world, but still no joy. I'm lingering on
>> irc for a bit before I head to bed if anyone has any ideas.
>
> Both (Jarod's and mine) are up again. My builder has stability problems
> since I upgraded to F12 on the host.

Mine was completely unstable with any kernel newer than 2.6.27.x, so
it was running a local 2.6.27.42 build, before it died. The machine
the guest has been transplanted to (for now) is chugging along just
fine with 2.6.32.3 though. The old machine was an 8-way opteron, the
current one is a core 2 quad. (Likely to move it one more time, either
back to the 8-way running RHEL5 as the host OS, or another machine).

> I have now again disabled all the
> virtio devices. I was then no longer able to obtain a dhcp address using
> libvirtd. I have now a static address in my builder and Thorsten
> restarted the main plague server and now finally both are available
> again.

I'm assuming the bounce of the main server was the key ingredient,
since I didn't change anything else on my end while I was asleep...

-- 
Jarod Wilson
jarod at wilsonet.com