Hi All, Just a note to people hosting builders, due to glibc changes in rawhide all builders now need run kernels >= 2.6.32, otherwise rawhide / development builds will likely fail.

On Fri, Jul 23, 2010 at 9:34 AM, Hans de Goede<j.w.r.degoede(a)gmail.com> wrote: > Hi All, > > Just a note to people hosting builders, due to glibc changes > in rawhide all builders now need run kernels>= 2.6.32, otherwise > rawhide / development builds will likely fail. That's... suboptimal. Pretty sure almost all the builders are currently el5-based. I know mine is.

Can you point out the kernel changes that are explicitly required here? I can look at backporting them for the rhel5 kernel, and assuming they aren't too invasive, don't break kabi, etc., I'm pretty sure I could talk the rhel5 kernel maintainer into taking 'em...

On Fri, Jul 23, 2010 at 10:00 AM, Hans de Goede <j.w.r.degoede(a)gmail.com&gt; wrote:

>>> Just a note to people hosting builders, due to glibc changes >>> in rawhide all builders now need run kernels>= 2.6.32, otherwise >>> rawhide / development builds will likely fail.

> AFAIK the current Fedora builders are running EL6 beta2 Yep, Dennis mentioned that to me the other day. He didn't mention exactly *why*, I just assumed we were eating our own dogfood... Guess I'll start coming up with a plan for updating my builder.

On Fri, Jul 23, 2010 at 6:45 PM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: > On Fri, Jul 23, 2010 at 10:14 AM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: >> On Fri, Jul 23, 2010 at 10:00 AM, Hans de Goede <j.w.r.degoede(a)gmail.com&gt; wrote: > ... >>>>> Just a note to people hosting builders, due to glibc changes >>>>> in rawhide all builders now need run kernels>= 2.6.32, otherwise >>>>> rawhide / development builds will likely fail. > ... >>> AFAIK the current Fedora builders are running EL6 beta2 >> >> Yep, Dennis mentioned that to me the other day. He didn't mention >> exactly *why*, I just assumed we were eating our own dogfood... Guess >> I'll start coming up with a plan for updating my builder. > > Shutting my builder vm down, copying the disk image, then starting a > rhel6 upgrade attempt... Wasn't exactly smooth, but the vm is up and running rhel6 now, and its reporting in to the build system... Flinging a build at it now... So far, so good.

On Sat, Jul 24, 2010 at 1:03 AM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: > On Sat, Jul 24, 2010 at 12:50 AM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: >> On Fri, Jul 23, 2010 at 6:45 PM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: >>> On Fri, Jul 23, 2010 at 10:14 AM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: >>>> On Fri, Jul 23, 2010 at 10:00 AM, Hans de Goede <j.w.r.degoede(a)gmail.com&gt; wrote: >>> ... >>>>>>> Just a note to people hosting builders, due to glibc changes >>>>>>> in rawhide all builders now need run kernels>= 2.6.32, otherwise >>>>>>> rawhide / development builds will likely fail. >>> ... >>>>> AFAIK the current Fedora builders are running EL6 beta2 >>>> >>>> Yep, Dennis mentioned that to me the other day. He didn't mention >>>> exactly *why*, I just assumed we were eating our own dogfood... Guess >>>> I'll start coming up with a plan for updating my builder. >>> >>> Shutting my builder vm down, copying the disk image, then starting a >>> rhel6 upgrade attempt... >> >> Wasn't exactly smooth, but the vm is up and running rhel6 now, and its >> reporting in to the build system... Flinging a build at it now... So >> far, so good. > > Spoke too soon. Fail: > > ... > DEBUG util.py:255: >      updates-released 137 k > DEBUG util.py:255:   xz-libs               x86_64 > 4.999.9-0.2.beta.20091007git.fc12 > DEBUG util.py:255: >      updates-released  89 k > DEBUG util.py:255:   xz-lzma-compat        x86_64 > 4.999.9-0.2.beta.20091007git.fc12 > DEBUG util.py:255: >      updates-released  15 k > DEBUG util.py:255:   zlib                  x86_64 1.2.3-23.fc12 >      fedora            69 k > DEBUG util.py:255:  Transaction Summary > DEBUG util.py:255: > ================================================================================ > DEBUG util.py:255:  Install     102 Package(s) > DEBUG util.py:255:  Upgrade       0 Package(s) > DEBUG util.py:255:  Total download size: 81 M > DEBUG util.py:255:  Installed size: 296 M > DEBUG util.py:255:  warning: /etc/hosts created as /etc/hosts.rpmnew > DEBUG util.py:255:  *** glibc detected *** /usr/bin/python: double > free or corruption (!prev): 0x000000000934f740 *** > DEBUG util.py:255:  ======= Backtrace: ========= > DEBUG util.py:328:  Child returncode was: -6 > DEBUG backend.py:593:  umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/proc > DEBUG util.py:289:  Executing command: umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/proc > DEBUG util.py:328:  Child returncode was: 0 > DEBUG backend.py:593:  umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/sys > DEBUG util.py:289:  Executing command: umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/sys > DEBUG util.py:328:  Child returncode was: 0 > DEBUG backend.py:593:  umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/dev/pts > DEBUG util.py:289:  Executing command: umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/dev/pts > DEBUG util.py:328:  Child returncode was: 0 > DEBUG backend.py:593:  umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/dev/shm > DEBUG util.py:289:  Executing command: umount -n > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c/root/dev/shm > DEBUG util.py:328:  Child returncode was: 0 > DEBUG util.py:77:  remove tree: > /var/lib/mock/fedora-12-x86_64-rpmfusion_free-5d5b7fcdebeb364e7600033c101fedcd3a63cd0c > DEBUG util.py:97:  kill orphans > > > I saw a similar glibc bomb running a yum remove while cleaning up some > detritus earlier tonight too. Looks like something bugly somewhere in > the rhel6 stack. Will look into it more tomorrow, its past my bed > time. :\ Red Hat Bugzilla #608710 and friends. I've applied a patch attached to that bug (patches an rpm lib) that is thought to possibly fix the problem and installed the resulting build on my builder, and just threw a previously failed build at it... Quite a few rhel6 kvm guest installs are apparently hitting this one, so I expect if this patch isn't the fix, someone is going to get to the bottom of this post-haste. :) (Fingers crossed the one-line patch is indeed the fix though).

On Tue, Jul 27, 2010 at 12:57 AM, Jarod Wilson <jarod(a)wilsonet.com&gt; wrote: ... >>> I saw a similar glibc bomb running a yum remove while cleaning up some >>> detritus earlier tonight too. Looks like something bugly somewhere in >>> the rhel6 stack. Will look into it more tomorrow, its past my bed >>> time. :\ >> >> Red Hat Bugzilla #608710 and friends. I've applied a patch attached to >> that bug (patches an rpm lib) that is thought to possibly fix the >> problem and installed the resulting build on my builder, and just >> threw a previously failed build at it... Quite a few rhel6 kvm guest >> installs are apparently hitting this one, so I expect if this patch >> isn't the fix, someone is going to get to the bottom of this >> post-haste. :) (Fingers crossed the one-line patch is indeed the fix >> though). > > Sigh. Never mind. Hadn't seen the latest few comments in the bug until > now. However, I have a new idea for something that may help (based on > some comments in bug 607650)... May well be an issue with transparent > hugepage support in kvm guests. I've turned that feature off now. Managed to make it all the way through populating two mock chroots simultaneously with transparent hugepage support disabled, pretty sure that's the answer, so please go ahead with any recent failed builds (I think all I saw was ffmpeg builds from Nicolas and xine-lib-extras-freeworld from Rex).