Getting blank outputs via API calls

[Shwetanjali]

Hello,

I have been able to start the OCP server, clients for Result Processing and Content Gathering on one of my organization’s server. I am trying to use the discovery feature of OCP for asset management. For the sake of exploring on dynamic discovery feature, I tried using the admin console and entered the network IP Range as mentioned on the page- https://www.opencontentplatform.org/get-started/. Then I have enabled the ‘Find’ jobs like find_PowerShell, find_PowerShell_test and also others features through Jobs-Modify-Toggle ribbon. I can see 63 endpoints connected. Following are the logs from ContentGatheringJobStatus-

2021-02-05 21:38:28.457047 info find_PowerShell_test, {‘data’: {‘address’: ‘192.168.0.38’, ‘realm’: ‘default’}, ‘class_name’: ‘IpAddress’, ‘identifier’: ‘4f1b8b31b9864b71b7fbe83d3daf9965’, ‘label’: ‘IpAddress’}, 2021-02-05 21:38:26.681070, 2021-02-05 21:38:28.457046, FAILURE, {‘New-PSSession : [192.168.0.38] Connecting to remote server 192.168.0.38 failed with the following error message : The\nWinRM client cannot process the request. Default authentication may be used with an IP address under the following\nconditions: the transport is HTTPS or the destination is in the TrustedHosts list, and explicit credentials are\nprovided. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be\nauthenticated. For more information on how to set TrustedHosts run the following command: winrm help config. For more\ninformation, see the about_Remote_Troubleshooting Help topic.\nAt line:1 char:6\n+ $s = New-PSSession -ComputerName “192.168.0.38” -Credential $ocpc; $?\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+ CategoryInfo : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin\ngTransportException\n+ FullyQualifiedErrorId : CannotUseIPAddress,PSSessionOpenFailed’}, {}

So certainly there is some error in the logs which I am not able to figure out how to resolve.
I followed the steps from Enable Dynamic Discovery https://www.opencontentplatform.org/get-started/ of updating the endpoint in shell_OS_create_config_group job and enabling it and turning it back off. However, I cannot see anything through Config groups ribbon and the output I get by running API(http://localhost:52707/ocp/job/runtime/contentGathering/shared.shell_OS_create_config_group)-
{
“Alle-MonSrv01”: {
“endpoint”: “Alle-MonSrv01”,
“status”: “FAILURE”,
“messages”: “[‘Could not find endpoint 192.168.0.38 with a correspoinding shell. Please make sure to run the corresponding \”Find\” job first.’]”,
“client_name”: “Alle-MonSrv01-1”,
“time_started”: “2021-02-05 03:51:09”,
“time_finished”: “2021-02-05 03:51:11”,
“time_elapsed”: 2.077645,
“result_count”: {},
“date_first_invocation”: “2021-02-05 01:14:14”,
“date_last_invocation”: “2021-02-05 03:51:09”,
“date_last_success”: null,
“date_last_failure”: “2021-02-05 03:51:09”,
“consecutive_jobs_passed”: 0,
“consecutive_jobs_failed”: 65,
“total_jobs_passed”: 0,
“total_jobs_failed”: 65,
“total_job_invocations”: 65
}
}

[codingadvocate]

Good to hear you have the platform up and running, and now basic jobs running.

The first error you showed is with the remote PowerShell connection to an endpoint (192.168.0.38). You can troubleshoot outside of OCP by opening PowerShell on the machine where you ran the contentGathering client, and issue a remote PS session to the target endpoint. You should see the same error as you showed above. This becomes a system administration type troubleshooting session, on whether the OS is enabled for PS remoting, whether to enable trusted IPs or DNS, etc.

You mentioned you had 63 other endpoints connected. I’d suggest using one of those IP’s for the config group job. You won’t be able to run that job on an endpoint that does not yet have a connected shell (like on 192.168.0.38), which is what that second error message says above (Could not find endpoint 192.168.0.38 with a corresponding shell. Please make sure to run the corresponding Find job first).

[Shwetanjali]

Hi,

The issue is not just one endpoint. For other endpoints either I am getting the above error or a WARNING.
This time instead of entering the entire IP range, I tried by adding just one IP-192.168.6.41.
Job “find_ICMP_socket_test” ran successfully.
2021-02-08 03:08:18.290569 info find_ICMP_socket_test, {‘value’: ‘192.168.6.41’}, 2021-02-08 03:08:18.270571, 2021-02-08 03:08:18.289568, SUCCESS, {‘IpAddress’: 1}
But with job “find_PowerShell_test” gave following WARNING.
2021-02-08 09:48:55.354664 info find_PowerShell_test, {‘data’: {‘address’: ‘192.168.6.41’, ‘realm’: ‘default’}, ‘class_name’: ‘IpAddress’, ‘identifier’: ‘909e306c6fa549dfb8111d027322563a’, ‘label’: ‘IpAddress’}, 2021-02-08 09:48:53.350975, 2021-02-08 09:48:55.354663, WARNING, {‘Endpoint not listening on ports [5985, 5986]; skipping PowerShell attempt.’}, {}

Hence, getting no results with API-http://localhost:52707/ocp/job/runtime/contentGathering/shared.shell_OS_create_config_group

For the machine 192.168.6.41, the admin has freed the ports 5985,5986 and also on the machine where OCP server is running. But still the warning persists.

[codingadvocate]

Take a look at the input parameters on the job. When the portTestBeforeConnectionAttempt is set to true, if the socket connection fails to verify the ports (listed in portsToTest) are open, then it doesn’t attempt a remote powershell session.

The current error says, “Endpoint not listening on ports [5985, 5986]; skipping PowerShell attempt.” And that’s what it did.

Your previous error message was powershell specific: “WinRM client cannot process the request. Default authentication may be used with an IP address under the following conditions…”. You will see the same thing if you test outside of OCP.

As I suggested before, this is a system administration task. Looks like you need to enable PowerShell (or PS Remoting) such that you can remotely connect from the contentGathering client machine – to those target Windows servers.

[Shwetanjali]

Hi,

I tried enabling PowerShell as you suggested and got this o/p – WinRM service is already running on this machine.\n WinRM is already set up for remote management on this computer.
But the logs still showed the same error.
I even tried running New-PSSession command for the same machine where contentGathering client(ip-192.168.0.85, name-Alle-MonSrv01) is running. On PowerShell, the output for the command- “New-PSSession -ComputerName Alle-MonSrv01” was
Id Name ComputerName ComputerType State ConfigurationName Availability
— —- ———— ———— —– —————– ————
19 WinRM19 Alle-MonSrv01 RemoteMachine Opened Microsoft.PowerShell Available

However, for the same machine, contentGatheringJobStatus logs showed-
2021-02-09 05:38:35.121707 info find_PowerShell_test, {‘data’: {‘address’: ‘192.168.0.85’, ‘realm’: ‘default’}, ‘class_name’: ‘IpAddress’, ‘identifier’: ‘79925c9ed19d4a48b2f059e94571bafc’, ‘label’: ‘IpAddress’}, 2021-02-09 05:38:33.982079, 2021-02-09 05:38:35.121706, FAILURE, {‘New-PSSession : [192.168.0.85] Connecting to remote server 192.168.0.85 failed with the following error message :\nAccess is denied. For more information, see the about_Remote_Troubleshooting Help topic.\nAt line:1 char:6\n+ $s = New-PSSession -ComputerName “192.168.0.85” -Credential $ocpc; $?\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+ CategoryInfo : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin\ngTransportException\n+ FullyQualifiedErrorId : AccessDenied,PSSessionOpenFailed’}, {}

[codingadvocate]

From what I can tell, you’re using name in the manual connection but using IP in the OCP connection. If you test with “-ComputerName 192.168.0.85” then you should see the message OCP is showing you.

To use IPs, make sure you have setup TrustedHosts to enable IPs. To use names, go to Platform->Configuration->OS Parameters in the admin console. Select Windows, and change the “connectViaFQDN” to true. OCP uses IPs by default since DNS isn’t always reliable at companies, but it can attempt name resolution if you tell it to.

Corporate IT usually has Group Policies or other automated means to globally set the PowerShell settings for PowerShell usage. But if you want to test it (again this is sys admin troubleshooting and not OCP troubleshooting – so here’s a freebie for you)… to setup IP-based connections:
====================================================
On Windows 10 Pro I just walked through this by disabling the Windows Firewall for both “Domain Profile” and “Private Profile”.

Then using Norton Security as my firewall, I added a port traffic rule to allow local port connections in on 5985 (HTTP) and 5986 (HTTPS) for PowerShell default ports.

To ensure the user’s group has rights to use powershell:
Set-PSSessionConfiguration -Name Microsoft.PowerShell -ShowSecurityDescriptorUI -Force

Right click on PowerShell and ‘Open as Administrator”:
> WinRM quickconfig -force
> Enable-PSRemoting
> Set-Item WSMAN:\Localhost\Client\TrustedHosts -Value “192.168.0.*” -Force

Then to view the Trusted hosts:
> Get-Item WSMan:\Localhost\Client\TrustedHosts
=======================================

And then to manually test:
=======================================
> $c = Get-Credential

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential

> $s = New-PSSession -ComputerName “192.168.0.85” -Credential $c

> Invoke-Command -Session $s -ScriptBlock {hostname}

<remote_server_name>

> Remove-PSSession $s
=======================================

[Shwetanjali]

Hi

I have followed each step as you suggested. I have set “connectViaFQDN” to true, then also changed the firewall settings as suggested.
Then on I opened powershell as admin and ran following commands-
PS C:\> WinRM quickconfig -force
WinRM service is already running on this machine.
WinRM is already set up for remote management on this computer.
PS C:\> Enable-PSRemoting
PS C:\> Set-Item WSMAN:\Localhost\Client\TrustedHosts -Value “192.168.0.*” -Force
PS C:\> Get-Item WSMan:\Localhost\Client\TrustedHosts

WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Client

Type Name SourceOfValue Value
—- —- ————- —–
System.String TrustedHosts 192.168.0.*

PS C:\> $c = Get-Credential

cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential
PS C:\> $s = New-PSSession -ComputerName “192.168.0.85” -Credential $c
PS C:\> Invoke-Command -Session $s -ScriptBlock {hostname}
Alle-MonSrv01
Still logs show the same result and I am not getting required o/p with APIs.

[Shwetanjali]

Since my OCP server and both ContentGatheringClient and ResultProcessingClient are running on same machine(192.168.0.85-Alle-MonSrv01), so I tried running this API “http://localhost:52707/ocp/config/client/Alle-MonSrv01&#8221; to get client information. This API is working fine as I got the required O/P. The issue is with “Find” jobs which I believe are necessary for doing asset management and I cannot figure out how to resolve this issue.
O/P from above API for reference-
{
“endpointName”: “Alle-MonSrv01”,
“platformType”: “Windows-10-10.0.17763-SP0”,
“platformSystem”: “Windows”,
“platformMachine”: “AMD64”,
“platformVersion”: “10.0.17763”,
“cpuType”: “Intel64 Family 6 Model 62 Stepping 4, GenuineIntel”,
“cpuCount”: “2”,
“memoryTotal”: “5 GB”,
“health”: {
“ContentGatheringClient”: [
{
“last_system_status”: “2021-02-10 02:42:38”,
“cpu_avg_utilization”: “42.1”,
“memory_aprox_total”: “5 GB”,
“memory_aprox_avail”: “490 MB”,
“memory_percent_used”: “92.0”,
“process_cpu_percent”: “0.0”,
“process_memory”: “101810176”,
“process_start_time”: “2021-02-10 00:43:36”,
“process_run_time”: “01:59:01”,
“object_id”: “0ce0e30b0d4f4a55ba4b6ed86cc289d6”,
“name”: “Alle-MonSrv01-1”
}
],
“ResultProcessingClient”: [
{
“last_system_status”: “2021-02-10 02:42:36”,
“cpu_avg_utilization”: “41.3”,
“memory_aprox_total”: “5 GB”,
“memory_aprox_avail”: “489 MB”,
“memory_percent_used”: “92.0”,
“process_cpu_percent”: “0.0”,
“process_memory”: “78688256”,
“process_start_time”: “2021-02-09 22:04:34”,
“process_run_time”: “04:38:01”,
“object_id”: “5660962e61e74f3ba6d96c57eef9a8f7”,
“name”: “Alle-MonSrv01-1”
}
],
“UniversalJobClient”: []
}
}

[codingadvocate]

To summarize:

1) the OCP platform is running
2) the OCP API responds as expected
3) you’re seeing errors on a contentGathering job that attempts to connect to endpoints via PowerShell
4) this requires system administration troubleshooting with how PowerShell is setup/enabled

If you just enabled firewalls/PS access to enable IPs, then reset “connectViaFQDN” to false and use IPs (the default).

Also, don’t try remotely connecting to the same system. Windows security policies (from about 2 years ago now) mostly prevents that type of activity – on purpose. So if you’re going to test externally, open a PS session on the 192.168.0.85 machine (or wherever you’re running the contentGathering client)… and issue the New-PSSession to a different target endpoint (different IP).

This is where my help stops on this topic. The logs will tell you what is happening, so leverage them and relay that to a sys admin. Feel free to open new posts when you find and fix the PowerShell setup.

[Author]

Posts