SQR-030: Jenkins Administration

  • Joshua Hoblitt

Latest Revision: 2020-02-13

1   How To

1.1   Clean up All Job Workspaces

Leeroy runs the jenkins-node-cleanup job periodically in order to automatically clean up job workspaces when the node on which they are located drops below a set free space threshold. In the event that there is stale state and/or corruption of workspaces, there is a job parameter named FORCE_CLEANUP that when set to true, will force workspaces on a node to be cleaned up regardless of the remaining free space.

Note

This is a big hammer. Use it wisely.

Note

jenkins-node-cleanup does not remove the workspace of running builds. As an example, if a stack-os-matrix build is running when jenkins-node-cleanup is triggered, even with FORCE_CLEANUP==true, the stack-os-matrix workspaces on active node(s) will not be disturbed.

Jenkins screenshot

Figure 1 Example of triggering a build with FORCE_CLEANUP==true:

1.2   Disable OSX jobs

To disable OSX jobs, you must comment out sections of build_matrix.yaml. Note that there are two secctions that need comments. Make sure not to miss commenting out the tarball builds as well.

Make a branch, then you can make your edits. File a PR, and when approved self merge or the reviewer can do so.

See the following screenshot from an example PR.

PR screenshot

Figure 2 Example diff of a PR for turning off OSX builds.

1.3   Update Conda Mirror

From time to time the upstream conda repos are in an consistent state during the LSST conda mirror update process. When this happens, the update-cmirror job should be manually triggered until the LSST mirror is in a consistent state. This may take more than once as it is dependent upon the upstream repos themselves first being fixed.