Foreman bug Warning Infinity

Yet another bug in newly installed Foreman,

While browsing diffrent parts of foreman gui, I get this error:

We're sorry, but something went wrong!

Warning!

Infinity

And in the logs:

# cat /var/log/foreman/production.log

Started GET "/hosts?search=last_report+%3E+%225+minutes+ago%22+and+status.enabled+%3D+true+and+status.applied+%3D+0+and+status.failed+%3D+0+and+status.pending+%3D+0" for 10.8.204.208 at 2015-06-26 14:53:12 +0200

2015-06-26 14:53:12 [I] Processing by HostsController#index as HTML

2015-06-26 14:53:12 [I]   Parameters: {"search"=>"last_report > \"5 minutes ago\" and status.enabled = true and status.applied = 0 and status.failed = 0 and status.pending = 0"}

2015-06-26 14:53:12 [I]   Rendered hosts/_list.html.erb (8.9ms)

2015-06-26 14:53:12 [I]   Rendered hosts/index.html.erb within layouts/application (12.8ms)

2015-06-26 14:53:12 [W] Operation FAILED: Infinity

2015-06-26 14:53:12 [I]   Rendered common/500.html.erb within layouts/application (4.1ms)

2015-06-26 14:53:12 [I]   Rendered layouts/base.html.erb (1.1ms)

2015-06-26 14:53:12 [I] Completed 500 Internal Server Error in 35ms (Views: 6.1ms | ActiveRecord: 0.0ms)

To solve this issue, the value for entries_per_page need to be set to a non 0 value:

Administer > Settings > General > entries_per_page 0 -> 20

mount: cannot remount block device /dev/mapper/vg00-lvvar read-write, is write-protected

Today one of our production systems filesystem /var went into read-only mode.

This is most probably because of underlying storage device issue, as we can see all the following errors in /var/log/messages:

Jul 2 12:08:53 hostname kernel: sd 2:0:2:0: [sdc] Unhandled error code
Jul 2 12:08:53 hostname kernel: sd 2:0:2:0: [sdc] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
Jul 2 12:08:53 hostname kernel: sd 2:0:2:0: [sdc] CDB: Write(10): 2a 00 68 c8 23 17 00 00 08 00
Jul 2 12:08:54 hostname kernel: Buffer I/O error on device dm-4, logical block 2163547
Jul 2 12:08:54 hostname kernel: lost page write due to I/O error on dm-4
Jul 2 12:08:54 hostname kernel: BUG: soft lockup - CPU#2 stuck for 67s! [udisks-daemon:10901]
Jul 2 12:08:54 hostname kernel: Modules linked in: bridge nfs lockd fscache auth_rpcgss nfs_acl sunrpc fuse autofs4 arc4 ecb md4 nls_utf8 cifs target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt 8021q garp stp llc ipv6 ext3 jbd uinput ppdev microcode vmware_balloon parport_pc parport e1000 sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Jul 2 12:08:54 hostname kernel: CPU 2
Jul 2 12:08:54 hostname kernel: Modules linked in: bridge nfs lockd fscache auth_rpcgss nfs_acl sunrpc fuse autofs4 arc4 ecb md4 nls_utf8 cifs target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt 8021q garp stp llc ipv6 ext3 jbd uinput ppdev microcode vmware_balloon parport_pc parport e1000 sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Jul 2 12:08:54 hostname kernel:
Jul 2 12:08:54 hostname kernel: Pid: 10901, comm: udisks-daemon Not tainted 2.6.32-431.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
Jul 2 12:08:54 hostname kernel: RIP: 0010:[<ffffffff8152a57e>] [<ffffffff8152a57e>] lock_kernel+0x2e/0x50
Jul 2 12:08:54 hostname kernel: RSP: 0018:ffff88021c987c88 EFLAGS: 00000206
Jul 2 12:08:54 hostname kernel: RAX: 0000000000000000 RBX: ffff88021c987c88 RCX: 0000000000000000
Jul 2 12:08:54 hostname kernel: RDX: 0000000000000001 RSI: ffff88021c987c48 RDI: ffff8802bb8a9ec0
Jul 2 12:08:54 hostname kernel: RBP: ffffffff8100bb8e R08: 0000000000000001 R09: ffff88021c987c48
Jul 2 12:08:54 hostname kernel: R10: 0000000000000002 R11: ffffffff81ac28d8 R12: ffffffff8119892f
Jul 2 12:08:54 hostname kernel: R13: ffff88021c987c48 R14: 0000000000000101 R15: ffff8802b6512cc0
Jul 2 12:08:54 hostname kernel: FS: 00007f8f0bae67a0(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
Jul 2 12:08:54 hostname kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 2 12:08:54 hostname kernel: CR2: 00007fb3fc537998 CR3: 000000021bffa000 CR4: 00000000000407e0
Jul 2 12:08:54 hostname kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 2 12:08:54 hostname kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 2 12:08:54 hostname kernel: Process udisks-daemon (pid: 10901, threadinfo ffff88021c986000, task ffff88021c8db540)
Jul 2 12:08:54 hostname kernel: Stack:
Jul 2 12:08:54 hostname kernel: ffff88021c987ce8 ffffffff811c5b5d 00000101282168d8 000000dd00000000
Jul 2 12:08:54 hostname kernel: <d> ffff88021c987cb8 ffff88021c831860 ffff88021c987cf8 ffff8802b5d578c0
Jul 2 12:08:54 hostname kernel: <d> 00000000fffffff4 ffff8802b9a9d0c0 ffffffff811c5f10 ffff8802b655d418
Jul 2 12:08:54 hostname kernel: Call Trace:
Jul 2 12:08:54 hostname kernel: [<ffffffff811c5b5d>] ? __blkdev_get+0x4d/0x3e0
Jul 2 12:08:54 hostname kernel: [<ffffffff811c5f10>] ? blkdev_open+0x0/0xc0


When I try to remount it rw, we get this error:

# mount -o rw,remount /var
mount: cannot remount block device /dev/mapper/vg00-lvvar read-write, is write-protected


and dmesg shows error:

# dmesg |grep -i mount
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user
EXT4-fs error (device dm-8): ext4_remount: Abort forced by user


To fix the issue, we need to reboot the server into rescue mode and do the “fsck” which will fix the issue but might loosing the data on that partition which in this case is /var. So we need to get a backup before proceeding with this option.