Wednesday, June 24, 2020

Trying to get stuff working again on the big processing cluster after its OS update has been like chasing a greased pig

And I'll be finding data that was not on the external raid system nor the long-term storage tapes for weeks I suppose.  Found one filetype yesterday that is going to be such a PITA to replace I may just not dink with it.  I really figured the cpu's internal raid would be fine after the OS update but nope, it got wiped out and the IT team couldn't find any way to restore it.

Then prima donna sr software engineer wanted the processing user's home area restored as well as all the normal user /home areas or at least his /home area.  Which wiped out the new .ssh area that allowed no password comms, kicking off jobs between the head node and the rest of the cluster so I had to ask for that to be restored by the IT team.   Then I was busy with other crap for about a week, went to tackle some processing last night and the slave nodes were missing a key system level rpm.  I did not catch that for a couple hours and was thinking I just didn't have a PATH env set correctly.   Hopefully now that IT has figured out that almost all the extra system level rpms needed for our stuff were installed only to the head node but not the slaves things will work tonight or tomorrow.   Turns out it was not just that one library file that I was getting the message for but 100s of rpms they have to get onto the slave nodes.  I sure hope the process goes much smoother when they go to start migrating our real-time cpu's to this new OS.

I finally got back to spraying some burrs this morning.  The burrs are not too horrible, definitely I need to spray but not so thick that I'm ready to go nuclear and spray everything with the mini boom sprayer.   The bad thing is there is such a lot of thistles.  I thought I had pulled up most of the thistles last year before they went to seed and would not have much issue this year but no such luck.


No comments: