Alerts from Solarwinds to show /WIP Partition is full
Posted May 4, 2021 12:58 AM
This is done in Cast IronAlert from Solarwinds to advise the /WIP partition is full. This will result in orchestrations not being processed if the volume fills up completely. This is likely caused by someone enabling debugging on one or more of the orchestrations. Only carry out these steps if the appliance is not processing.
- Locate the password for the CastIron appliance in PasswordState.
- Connect to the Cast Iron appliance (If you're able to login to the Cast Iron appliance using the web management interface proceed with steps shown below under 2a; If you're not able to to the Cast Iron appliance using the web management interface, see step 2b below)
- Navigate to the System → Commands menu
- Select command Remove job history in the drop-down list and press Go (this is equivalent to the console command system clean orchmon) Running this command will cause the appliance to restart.
![[Image: 01e5ab04ed9f3269547f4cd4f586437e.png]](https://gyazo.com/01e5ab04ed9f3269547f4cd4f586437e.png)
- After the appliance has restarted, login again and navigate to the System → Commands menu
- Select command Remove all jobs in progress in the drop-down list and press Go (this is equivalent to the console command system clean running) Running this command will cause the appliance to restart.
![[Image: fafeabcefc3d74681fe029176c239754.png]](https://gyazo.com/fafeabcefc3d74681fe029176c239754.png)
- After the appliance has restarted, login again and navigate to the System → Commands menu
- Select command Reboot in the drop-down list and press Go ((note this is the Reboot command NOT Restart) Running this command will cause the appliance to reboot.
![[Image: 2b1765a31448726b5a5ae00fd3db996b.png]](https://gyazo.com/2b1765a31448726b5a5ae00fd3db996b.png)
- After the appliance has rebooted, proceed to step 3 below
- Navigate to the System → Commands menu
- Browse to https://name of the appliance.
- Log on with the same credentials.
- Navigate to System > Hardware Status.
- Verify that the Disk utilisation has dropped back down to <10%.
- Navigate to Home.
- Verify that no orchestrations are in a Stopping or Stopped state. Running or Undeployed is fine. Keep an eye on anything in a state of Starting and make sure they start properly. If not, escalate to the IT Application Services team.
- For abn-ci3, due to dependency of the processes with SAP, following processes need to be stopped, undeployed and start them all in one go
![[Image: a1c60551306468da279a2548a66bb1fd.png]](https://gyazo.com/a1c60551306468da279a2548a66bb1fd.png)
- Finally, locate and reduce the logging level on the orchestrations, to prevent the partition from filling up again.
- For abn-ci3, due to dependency of the processes with SAP, following processes need to be stopped, undeployed and start them all in one go
Reducing logging levels
- Navigate to Repository > Configurations.
- Select each orchestration in turn until you find one where Logging Level is set to anything other than "Error Values". Record the name of this orchestration.
![[Image: 4bd8be91e7e23d1e44916c23ea392e25.png]](https://gyazo.com/4bd8be91e7e23d1e44916c23ea392e25.png)
- Click Edit in the Orchestrations field.
- Use the drop-down menu to select "Error Values".
- Press Save.
