Home Upgrade Search Memberlist Extras Hacker Tools Award Goals Help Wiki Follow Contact

HF Rulez the UniverseHF Rulez the Universe
Sigma
View all of my HELPFUL THREADS!!
Tutorials Solarwinds vulnerability Cast Iron

Alerts from Solarwinds to show /WIP Partition is full

Posted 05-04-2021, 12:58 AM
This is done in Cast Iron

Alert from Solarwinds to advise the /WIP partition is full. This will result in orchestrations not being processed if the volume fills up completely. This is likely caused by someone enabling debugging on one or more of the orchestrations. Only carry out these steps if the appliance is not processing.

  1. Locate the password for the CastIron appliance in PasswordState.
  2. Connect to the Cast Iron appliance (If you're able to login to the Cast Iron appliance using the web management interface proceed with steps shown below under 2a; If you're not able to to the Cast Iron appliance using the web management interface, see step 2b below)
    • Navigate to the SystemCommands menu
    • Select command Remove job history in the drop-down list and press Go (this is equivalent to the console command system clean orchmon) Running this command will cause the appliance to restart.
      [Image: 01e5ab04ed9f3269547f4cd4f586437e.png]
    • After the appliance has restarted, login again and navigate to the System Commands menu
    • Select command Remove all jobs in progress in the drop-down list and press Go (this is equivalent to the console command system clean running) Running this command will cause the appliance to restart.
      [Image: fafeabcefc3d74681fe029176c239754.png]
    • After the appliance has restarted, login again and navigate to the System Commands menu
    • Select command Reboot in the drop-down list and press Go ((note this is the Reboot command NOT Restart) Running this command will cause the appliance to reboot.
      [Image: 2b1765a31448726b5a5ae00fd3db996b.png]
    • After the appliance has rebooted, proceed to step 3 below
  3. Browse to https://name of the appliance.
  4. Log on with the same credentials.
  5. Navigate to System > Hardware Status.
  6. Verify that the Disk utilisation has dropped back down to <10%.
  7. Navigate to Home.
  8. Verify that no orchestrations are in a Stopping or Stopped state. Running or Undeployed is fine. Keep an eye on anything in a state of Starting and make sure they start properly. If not, escalate to the IT Application Services team.
    • For abn-ci3, due to dependency of the processes with SAP, following processes need to be stopped, undeployed and start them all in one go
      [Image: a1c60551306468da279a2548a66bb1fd.png]
    • Finally, locate and reduce the logging level on the orchestrations, to prevent the partition from filling up again.

Reducing logging levels
  1. Navigate to Repository > Configurations.
  2. Select each orchestration in turn until you find one where Logging Level is set to anything other than "Error Values". Record the name of this orchestration.
    [Image: 4bd8be91e7e23d1e44916c23ea392e25.png]
  3. Click Edit in the Orchestrations field.
  4. Use the drop-down menu to select "Error Values".
  5. Press Save.