This article details how to generate report using the RestAPI which will mirror as closely as possible the "OutputDevices" report which was available in V3 of iQSonar.
You can download the latest version of the file: OutputDevices-v1.ps1, and simply customise the $user, $pass and $sonar values and it should run in your environment
Fields in the original report
This information is taken from the iQSonar Developer API guide for iQSonar V3
Field | V3 Comment | RestAPI Comment |
---|---|---|
Hostname | Hostname of the device | Get this from the /devices RestAPI endpoint |
FQDN | Fully QUalified Domain Name of the device | Get this from the /devices/{Device_ID} RestAPI endpoint |
OS | OS description and service pack | Get this from the /devices/{Device_ID} RestAPI endpoint |
OS Install Date | OS description and service pack | Get this from the /devices/{Device_ID} RestAPI endpoint May not always be available |
Location | Location of the device | In V4 this represents the target configuration set used to define the device. Get this from the /devices RestAPI endpoint |
Serial Number | The serial number of the device | Get this from the /devices RestAPI endpoint |
PhysicalCPUCount | Number of physical CPUs or sockets in the Physical Device (which may differ from the Virtual or Logical host when a device is a virtual machine or partition) | IF the device is a physical device, this information is derived from the /devices/{Device_ID} RestAPI endpoint Where the virtualisation host is not scanned, then this field and all the other information about the physical host will not be able to be populated. |
PhysicalCoreCount | Sum of cores across all CPUs in the Physical Device (which may differ from the Virtual or Logical host when a device is a virtual machine or partition) | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalCoresPerCPU | Physical core count divided by the physical CPU count | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalCPUManufacturer | Manufacturer of the CPU on the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalCPUModel | Model of the CPU on the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalCPUSpeed | Speed of the CPU on the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalRAM | Total amount of RAM on the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
VirtualCPUCount | Number of Virtual CPUs perceived by the device (only populated when the device is Virtual or Logical) | Get this from the /devices/{Device_ID} RestAPI endpoint; Only available if device is virtual or logical |
VirtualCoreCount | Sum of Cores perceived by the device (only populated when the device is Virtual or Logical) | Get this from the /devices/{Device_ID} RestAPI endpoint; Only available if device is virtual or logical |
VirtualRAM | Amount of perceived RAM by the device (only populated when the device is Virtual or Logical) | Get this from the /devices/{Device_ID} RestAPI endpoint; Only available if device is virtual or logical |
DeviceModel | Model of the device | Get this from the /devices RestAPI endpoint for a physical device. |
PhysicalModelSocketCount | Number of sockets that may be populated with physical CPUs on the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalModelCoreCount | Maximum number of cores per CPU according to the model documentation | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalDeviceManufacturer | Manufacturer of the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalHostname | Hostname of the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint for virtual devices; May not always be available |
PhysicalFQDN | Fully Qualified Domain Name of the Physical Device | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
IP Address |
| V4 also exposes IP6 addresses. |
PhysicalMACAddress | If more than 1, will be semi-colon separated. | |
VirtualMACAddress | If more than 1, will be semi-colon separated. | |
ClusterInformation | The virtualization cluster that the physical machine is part of | Only available if it is part of a cluster, and the cluster has been scanned |
ClusterName | Name of cluster | Only available if it is part of a cluster, and the cluster has been scanned |
PartitioningMethod | Virtualization Method (VMware, HyperV, LPAR etc.). If this field is null it indicates the device is physical. | |
DerivedCPU | Not directly available via RestAPI. Must be coded in the script. | |
DerivedCoresPerCPU | Not directly available via RestAPI. Must be coded in the script. | |
Bios | Concatenation of Device.BIOSName, DeviceBIOSManufacturer, Device.BIOSVersion separated by semi-colons. | Get this from the /devices/{Device_ID} RestAPI endpoint. |
LastScanDate | Last date the device was scanned | Get this from the /devices/{Device_ID} RestAPI endpoint. |
DeviceID | The unique identifier for this device | The Rest API gives unique identifiers in GUID format |
PhysicalDeviceID | The unique identifier of the Physical Device (where relevant) – used to map logical devices or Virtual machines back to the Physical Device which hosts them. | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalCPUNotes | Notes that indicate CPU vs socket mismatches or CPU core values that don’t match the CPU model | This V3 data is not directly available via RestAPI. This functionality is part of DataHub for V4, and if required must be coded in the script. |
Notes | Notes related to the device model | This V3 data is not directly available via RestAPI. This functionality is part of DataHub for V4, and if required must be coded in the script. |
ExternalLink | Link to model documentation from the vendor | This V3 data is not available via RestAPI. |
DNSHostname | The hostname of the device as reported from DNS | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
DNSFQDN | The fully qualified hostname of the device as reported from DNS | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalDNSHostname | The hostname of the physical device as reported from DNS | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
PhysicalDNSFQDN | The fully qualified hostname of the physical device as reported from DNS | Get this from the /devices/{Device_ID} RestAPI endpoint; May not always be available |
MeasurementComment | Will contain additional info such as “Believed to be: Oracle Linux Server release 5.7” where appropriate | This V3 data is not available via RestAPI. |
Connect to the RestAPI
The RestAPI uses HTTP basic authentication rather than domain credentials. To write a script that extracts data from the RestAPI you will need to know the login name and password for an iQSonar user who has the "access Rest API" permission enabled. By default the admin user always has this permission. These variables will need to be configured for your site:
$user = "admin" $pass = "password" $sonar = "iQSonar Host" $secpass = ConvertTo-SecureString $pass -AsPlainText -Force $credential = New-Object System.Management.Automation.PSCredential($user,$secpass)
The first step is to determine exactly how many devices we're going to be collecting data for. This information is returned in the header of the RestAPI return value. By default a call to the devices endpoint will return a batch of 200 devices, and the user is expected to page through the list of available devices by making successive calls to this endpoint using an offset parameter to specify how many devices have already been seen. The fetch_size parameter is used to determine how many devices are returned per batch. The fastest way to determine the total number of available devices is to explicitly request the first device only. Since we want data from the headers of the result, we use the Invoke-WebRequest PowerShell cmdlet to get this information.
$uri = -join ("http://", $sonar, "/api/v1/devices/?offset=1&fetch_size=1") $r = Invoke-WebRequest $uri -Credential $credential # $r.headers has HTML headers, $r.content has text content $deviceCount = $r.headers.'X-fetch-count'
Build the CSV file header row
PowerShell has a number of built in libraries for handling output to various file formats. For this example we will be saving the results in a CSV file that can be viewed directly in EXCEL or imported into other databases.
Note that one limitation of using the PowerShell System.Object to build the CSV file is that the order of the colums is not fixed.
# Build the CSV File header row $csv = @() $row = New-Object System.Object $row | Add-Member -MemberType NoteProperty -Name "Hostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "OS" -Value $null $row | Add-Member -MemberType NoteProperty -Name "OS-Install-Date" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Location" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Serial-Number" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalRAM" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualCPUCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualCoreCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualRAM" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DeviceModel" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelSocketCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelCoreCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceManufacturer" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalFQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalMACAddress" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualMACAddress" -Value $null $row | Add-Member -MemberType NoteProperty -Name "ClusterInformation" -Value $null $row | Add-Member -MemberType NoteProperty -Name "ClusterName" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PartitioningMethod" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DerivedCPU" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DerivedCoresPerCPU" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Bios" -Value $null $row | Add-Member -MemberType NoteProperty -Name "LastScanDate" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DeviceID" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceID" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUNotes" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Notes" -Value $null $row | Add-Member -MemberType NoteProperty -Name "ExternalLink" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DNSHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSFQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "MeasurementComment" -Value $null
Process the list of devices
In the RestAPI the "devices" endpoint returns some summary information about each device and a link to the rest of the information about the device.
In an estate with fewer than about 500 devices scanned, you may well find that the most efficient way to process the list of devices is to pull all of the the summary information in one go. You can specify this by setting "fetch_size=all", and only use one outer loop. However in larger estates we would expect that a fetch_size of between 100 and 500 would produce best performance, and the default of 200 devices at a time is a good place to start. To demonstrate, rather than populating the CSV file, let us just print out the host name of each device seen. We use the Invoke-RestMethod powershell command to get the results in JSON format and convert them automatically to a PowerShell object.
Trivial processing - get the host names
$fs = 100 # fetch_size $offset = 1 # offset $seen = 1; # first offset is 1, not 0 while ( $seen -lt $devicecount) { # Outer loop, grab a batch (defined by fetch_size) of devices $uri = -join ("http://", $sonar, "/api/v1/devices/?offset=", $offset, "&fetch_size=", $fs) $devices = Invoke-RestMethod $url -Credential $credential $i = 1 while ($i -lt $devices.count) { # inner loop - process individual devices in this batch # Example Only $str = -join( "Hostname is: ", $devices[$i].host_name) write-host $str $i = $i + 1; # keep track for inner loop $seen = $seen + 1; # keep track for outer loop } # Finished this batch $offset = $seen }
Slightly more advanced, get all information available from the devices endpoint
Of the required fields, we can populate a small number of them directly from the devices endpoint without delving any deeper into the RestAPI. This includes the Hostname, the Serial Number, the DeviceID, and the Location. In many cases we can also get the DNS FQDN.
This code fragment also demonstrates dealing with missing values. In the JSON returned by the RestAPI, if the device is missing certain information the key-value pairs are simply not returned in most cases. Your code needs to handle this gracefully.
In other cases (for example qualified_name) there can be an array of values returned in no particular order. You need to loop over the array looking for the value you're interested in (e.g. FQDN or DNSFQDN) and ignoring the values you are not interested in (e.g. UnixName)
$fs = 100 # fetch_size $offset = 1 # offset $seen = 1; # first offset is 1, not 0 while ( $seen -lt $devicecount) { # Outer loop, grab a batch (defined by fetch_size) of devices $url = -join ("http://", $sonar, "/api/v1/devices/?offset=", $offset, "&fetch_size=", $fs) $devices = Invoke-RestMethod $url -Credential $credential $i = 1 while ($i -lt $devices.count) { # inner loop - process individual devices in this batch # We want to populate some of the data fields in the CSV file - what can we fill in just from the DEVICES end point $row = New-Object System.Object $row | Add-Member -MemberType NoteProperty -Name "DeviceID" -Value $devices[$i].device_id # If the path was only partly successful we might not have a serial number. However we can safely store the empty string. # if you are processing RestAPI using PHP or a different scripting language that is stricter about handling undefined variables, # you need error handling code around here too. $row | Add-Member -MemberType NoteProperty -Name "Serial-Number" -Value $devices[$i].serial_number # For certain devices, for example some SNMP devices, or for storage devices we might not get a hostname! Explicitly call out a device with no hostname. # Also if hostname is not defined, there will be no qualified name(s) if (!$devices[$i].host_name) { # Hostname is not defined $row | Add-Member -MemberType NoteProperty -Name "Hostname" -Value "(undefined)" $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value "(undefined)" $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value "(undefined)" } else { # Hostname is defined $row | Add-Member -MemberType NoteProperty -Name "Hostname" -Value $devices[$i].host_name # If the array is not present, $num gets value 0, and so the inner loop will not fire # we have already set FQDN and DNSFQDN to "(undefined)" where there is no hostname, but we can sometimes have a hostname but no FQDN $num = $devices[$i].qualified_name.count if ($num -eq 0) { $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value "(undefined)" $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value "(undefined)" } else { $j = 0 $fqdn = "(No FQDN)" while ($j -lt $num) { if ($devices[$i].qualified_name[$j].name_type -eq "DNSFQDN") { # DNSFQDN can also be used for FQDN. $fqdn = $devices[$i].qualified_name[$j].name $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value $devices[$i].qualified_name[$j].name $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value $devices[$i].qualified_name[$j].name } elseif ($devices[$i].qualified_name[$j].name_type -eq "FQDN") { $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value $devices[$i].qualified_name[$j].name } $j = $j + 1 } } } # Locations is an array containing one or more locations. We can store locations[0].name as the location $row | Add-Member -MemberType NoteProperty -Name "Location" -Value $devices[$i].locations[0].name # The Last Scan Date should always be there, but if absent PowerShell will happily record a NULL value for us. $row | Add-Member -MemberType NoteProperty -Name "LastScanDate" -Value $devices[$i].last_scan $csv += $row $i = $i + 1; # keep track for inner loop $seen = $seen + 1; # keep track for outer loop } # Finished this batch $offset = $seen }
Providing feedback to the user
Once we start iterating down into subsequent API calls for individual devices, the script will run for much longer than when we limit ourselves to the top level data. Therefore we need to provide some feedback so that the user does not assume the script has crashed. When we have finally processed all of the data, we can report to the user that we are finished, and save the CSV file.
# # To find out how many devices there are, ask for the first device and look at the header for the device count # offset=1 means start with first device. fetch_size=1 means only return one record. $uri = -join ("http://", $sonar, "/api/v1/devices/?offset=1&fetch_size=1") $r = Invoke-WebRequest $uri -Credential $credential # $r.headers has HTML headers, $r.content has text content $deviceCount = $r.headers.'X-fetch-count' # Let the user know how many devices we can see $output = -join ( "There are: ", $deviceCount, " devices in total") write-host $output # # ... set up CSV # $fs = 100 # fetch_size $offset = 1 # offset $seen = 1; # first offset is 1, not 0 while ( $seen -lt $devicecount) { # Outer loop, grab a batch (defined by fetch_size) of devices $url = -join ("http://", $sonar, "/api/v1/devices/?offset=", $offset, "&fetch_size=", $fs) $devices = Invoke-RestMethod $url -Credential $credential $i = 1 while ($i -lt $devices.count) { # inner loop - process individual devices in this batch # do something with the data $csv += $row $i = $i + 1; # keep track for inner loop $seen = $seen + 1; # keep track for outer loop if ( $seen % 10 -eq 0) { # progress indicator - display a "." every 10 devices write-host "." -nonewline } } # Finished this batch $offset = $seen } write-host " Done. Saving output to OutputDevices.csv now." # Save the output to a file $csv | Export-csv OutputDevices.csv -NoTypeInformation
Getting the rest of the info
There is a lot more information saved about each device which is not returned in the summary provided by the devices endpoint.
The way to access this information is by using the additional endpoints defined by devices/device-id
This endpoint in turn contain links to more endpoints. As of Gwynn R3, these endpoints are:
- devices/device-id/applications
- devices/device-id/components
- devices/device-id/installed_software
For the purposes of the OutputDevices report, all the information we are going to use is in the devices/device-id endpoint. A link to this endpoint us returned in the summary data in the "self" variable.
# Get extra data about the host $currDevice = Invoke-RestMethod $devices[$i].self -Credential $credential
Some of the data returned by this second invocation can be stored directly into the CSV fields, for example BIOS information or OS Install date, as long as it is present:
$row | Add-Member -MemberType NoteProperty -Name "OS" -Value $currDevice.operating_system.name if (!$currDevice.os_install_date) { $row | Add-Member -MemberType NoteProperty -Name "OS-Install-Date" -Value "(not available)" } else { $row | Add-Member -MemberType NoteProperty -Name "OS-Install-Date" -Value $currDevice.os_install_date } if (!$currDevice.bios.name) { $row | Add-Member -MemberType NoteProperty -Name "Bios" -Value $null } else { $row | Add-Member -MemberType NoteProperty -Name "Bios" -Value $currDevice.bios.name }
Some of the data requires additional logic. For example we might have no IP address (for a SAN datastore attached to a VMWare vCenter), we might have exactly one IP address, or we might have multiple addresses (for example a node in an Oracle RAC)
# IP Address is an array of 0 or more IP Addresses. We might not have any, or we might have LOTS. $num = $currDevice.ip_address.count if ($num -eq 0) { $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value "(no IP address)" } else { if ($num -eq 1) { $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value $currDevice.ip_address[0].ip_address } else { # list of addresses, needs to be separated by semi-colons $addresslist = "" $j = 0 while ($j -lt $num) { $addresslist = -join($addresslist, $currDevice.ip_address[$j].ip_address) if ($j -lt $num) { $addresslist = -join($addresslist, ";") } $j = $j + 1 } $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value $addresslist } }
Additional details for physical hosts
# Is this device physical or virtual if ($currDevice.is_virtual -eq "false") { # This is a physical target # We can fill in physical hardware details directly, and leave the entries for virtual hardware blank # Physical device RAM $row | Add-Member -MemberType NoteProperty -Name "PhysicalRAM" -Value = total_memory_mb # Physical Device CPU Info # Sometimes we don't get CPU Information if ( ($currDevice.cpu_count -eq 0) -or (!$currDevice.cpu_count) ) { # Physical CPU info not avialble $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" } else { # we have at least some CPU info $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value $currDevice.cpu_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value $currDevice.core_count # Further details will be derived from cpu[0] if present if (!$currDevice.cpu[0].manufacturer) { # additional info missing $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" } else { $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value $currDevice.cpu[0].core_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value $currDevice.cpu[0].manufacturer $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value $currDevice.cpu[0].cpu_model $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value $currDevice.cpu[0].cpu_speed } } $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelSocketCount" -Value "n/a" # not exposed in V4 RestAPI $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelCoreCount" -Value "n/a" # not exposed in V4 RestAPI $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceManufacturer" -Value $currDevice.manufacturer $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value $devices[$i].host_name # # Add this later # $row | Add-Member -MemberType NoteProperty -Name "PhysicalMACAddress" -Value $null }
Additional details for virtual hosts
For virtual hosts, we will have virtual device info corresponding to the physical device info stored above.
Additionally if iQSonar has scanned the virtualization host and identified the host that the VM is running on, we will also have the details for the host server and a pointer to any cluster that the host might be part of.
else { # This is a virtual target if ( ($currDevice.cpu_count -eq 0) -or (!$currDevice.cpu_count) ) { # Physical CPU info not avialble $row | Add-Member -MemberType NoteProperty -Name "VirtualCPUCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "VirtualCoreCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" } $row | Add-Member -MemberType NoteProperty -Name "VirtualCPUCount" -Value $currDevice.cpu_count $row | Add-Member -MemberType NoteProperty -Name "VirtualCoreCount" -Value $currDevice.core_count $row | Add-Member -MemberType NoteProperty -Name "VirtualRAM" -Value = $currDevice.total_memory_mb $row | Add-Member -MemberType NoteProperty -Name "DeviceModel" -Value $currDevice.model # Do something with MAC addresses for VMs # Now, if the VM host we're running on was scanned, we can do something with the following: if (!$currDevice.virtual_host.device_id) { # host not scanned, record an error by putting "n/a" in physical hostname, leave everything else blank $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value "n/a" } else { # host was scanned, so let's populate as many details as possible about the physical host $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value $currDevice.virtual_host.cpu_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value $currDevice.virtual_host.core_count if ( ($currDevice.virtual_host.cpu_count -ne 0) -and ($currDevice.virtual_host.core_count -ne 0)) { $cpcpu = $currDevice.virtual_host.core_count / $currDevice.virtual_host.cpu_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value $cpcpu } # In order to get the CPU Manufacturer information, you need to follow the link to # $currDevice.virtual_host.self and parse the CPU info there. This is left as an exercise to the user $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalRAM" -Value $currDevice.virtual_host.total_memory_mb $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceManufacturer" -Value $currDevice.virtual_host.manufacturer $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value $currDevice.virtual_host.host_name # In order to get the MAC Address for the physical host, you need to follow the link to # $currDevice.virtual_host.self and parse the MAC info there. This is left as an exercise to the user $row | Add-Member -MemberType NoteProperty -Name "PhysicalMACAddress" -Value "n/a" } }
But... we want all the columns in the same order as the original report
The script above works to extract the data, but if you want to feed the CSV file to a data processing application which was written for a V3 datasource, you probably want to have the columns in the "correct" order (including empty columns) so that your third party application does not need to be reconfigured or re-written. So, to force all the columns to be saved, including the columns that are always empty, and to have the columns in the correct order corresponding to the original V3 file, we use the Select-Object powershell cmdlet, and pipe the results into the Export-CSV cmdlet.
This produces a very long line of code, so we use backticks to break it up for readibility
$csv | Select-Object Hostname, FQDN, OS, OS-Install-Date, Location, Serial-Number, PhysicalCPUCount, PhysicalCoreCount, PhysicalCoresPerCPU, ` PhysicalCPUManufacturer, PhysicalCPUModel, PhysicalCPUSpeed, PhysicalRAM, VirtualCPUCount, VirtualCoreCount, VirtualRAM, DeviceModel, ` PhysicalModelSocketCount, PhysicalModelCoreCount, PhysicalDeviceManufacturer, PhysicalHostname, PhysicalFQDN, IP-Address, ` PhysicalMACAddress, VirtualMACAddress, ClusterInformation, ClusterName, PartitioningMethod, DerivedCPU, DerivedCoresPerCPU, ` Bios, LastScanDate, DeviceID, PhysicalDeviceID, PhysicalCPUNotes, Notes, ExternalLink, DNSHostname, DNSFQDN, PhysicalDNSHostname, ` PhysicalDNSFQDN, MeasurementComment | Export-csv OutputDevices.csv -NoTypeInformation
What have we left out?
This sample code will work as is, but deliberately it omits a few details.
- MAC Addresses - these can be found in devices/device_id along with the IP addresses and processed in the same way. The v3 report distinguishes between MAC addresses assigned to VMs and to Physical Hosts. Also, in the case of determining the MAC address of the physical host for a VM, this can be found by making a call from devices/device_id/self → virtual_host.self This call was omitted from the sample code as an additional RestAPI call imposes a performance penalty if it is not needed.
- Cluster Information - this can be found by making a call from devices/device_id/self → virtual_host.self This call was omitted from the sample code as an additional RestAPI call imposes a performance penalty if it is not needed.
- PartitioningMethod - this needs to be derived from the OS of the physical host, or the "manufacturer" for a VM. iQSonar supports scanning VMWare VCenters, Microsoft HyperV hypervisors, Solaris Zones and IBM LPARs.
- Physical CPU details for VMs - this can be found by making a call from devices/device_id/self → virtual_host.self This call was omitted from the sample code as an additional RestAPI call imposes a performance penalty if it is not needed.
- The DerivedCPU and DerivedCoresPerCPU columns are blank. In V3 these values were "cleaned" from the CPU information. This can be done as a post-processing step for the CSV file if needed.
Putting it all together
This is the script so far. It gathers all the available data from the devices and devices/device_id endpoints, and saves it as a CSV file. The order of the columns in the CSV file is non-deterministic. (It is set by the order of insertion of the columns of the last item in the dataset.)
# # PowerShell Worked Examples # # Generate OutputDevices equivalent report via RestAPI # # The RestAPI uses HTTP authentication. Set $user and $pass to the username and password needed for your iQSonar instance # set the $host to the IPAddress, name or FQDN of the iQSonar server $user = "admin" $pass = "password" $sonar = "iQSonar Host" $secpass = ConvertTo-SecureString $pass -AsPlainText -Force $credential = New-Object System.Management.Automation.PSCredential($user,$secpass) # # To find out how many devices there are, ask for the first device and look at the header for the device count # offset=1 means start with first device. fetch_size=1 means only return one record. $uri = -join ("http://", $sonar, "/api/v1/devices/?offset=1&fetch_size=1") $r = Invoke-WebRequest $uri -Credential $credential # $r.headers has HTML headers, $r.content has text content $deviceCount = $r.headers.'X-fetch-count' # Let the user know how many devices we can see $output = -join ( "There are: ", $deviceCount, " devices in total") write-host $output # Build the CSV File header row $csv = @() $row = New-Object System.Object $row | Add-Member -MemberType NoteProperty -Name "Hostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "OS" -Value $null $row | Add-Member -MemberType NoteProperty -Name "OS-Install-Date" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Location" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Serial-Number" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalRAM" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualCPUCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualCoreCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualRAM" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DeviceModel" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelSocketCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelCoreCount" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceManufacturer" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalFQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalMACAddress" -Value $null $row | Add-Member -MemberType NoteProperty -Name "VirtualMACAddress" -Value $null $row | Add-Member -MemberType NoteProperty -Name "ClusterInformation" -Value $null $row | Add-Member -MemberType NoteProperty -Name "ClusterName" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PartitioningMethod" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DerivedCPU" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DerivedCoresPerCPU" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Bios" -Value $null $row | Add-Member -MemberType NoteProperty -Name "LastScanDate" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DeviceID" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceID" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUNotes" -Value $null $row | Add-Member -MemberType NoteProperty -Name "Notes" -Value $null $row | Add-Member -MemberType NoteProperty -Name "ExternalLink" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DNSHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSFQDN" -Value $null $row | Add-Member -MemberType NoteProperty -Name "MeasurementComment" -Value $null $fs = 100 # fetch_size $offset = 1 # offset $seen = 1; # first offset is 1, not 0 while ( $seen -lt $devicecount) { # Outer loop, grab a batch (defined by fetch_size) of devices $url = -join ("http://", $sonar, "/api/v1/devices/?offset=", $offset, "&fetch_size=", $fs) $devices = Invoke-RestMethod $url -Credential $credential $i = 1 while ($i -lt $devices.count) { # inner loop - process individual devices in this batch # We want to populate some of the data fields in the CSV file - what can we fill in just from the DEVICES end point $row = New-Object System.Object $row | Add-Member -MemberType NoteProperty -Name "DeviceID" -Value $devices[$i].device_id # If the path was only partly successful we might not have a serial number. However we can safely store the empty string. # if you are processing RestAPI using PHP or a different scripting language that is stricter about handling undefined variables, # you need error handling code around here too. $row | Add-Member -MemberType NoteProperty -Name "Serial-Number" -Value $devices[$i].serial_number # For certain devices, for example some SNMP devices, or for storage devices we might not get a hostname! Explicitly call out a device with no hostname. # Also if hostname is not defined, there will be no qualified name(s) if (!$devices[$i].host_name) { # Hostname is not defined $row | Add-Member -MemberType NoteProperty -Name "Hostname" -Value "(undefined)" $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value "(undefined)" $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value "(undefined)" } else { # Hostname is defined $row | Add-Member -MemberType NoteProperty -Name "Hostname" -Value $devices[$i].host_name # If the array is not present, $num gets value 0, and so the inner loop will not fire # we have already set FQDN and DNSFQDN to "(undefined)" where there is no hostname, but we can sometimes have a hostname but no FQDN $num = $devices[$i].qualified_name.count if ($num -eq 0) { $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value "(undefined)" $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value "(undefined)" } else { $j = 0 $fqdn = "(No FQDN)" while ($j -lt $num) { if ($devices[$i].qualified_name[$j].name_type -eq "DNSFQDN") { # DNSFQDN can also be used for FQDN. $fqdn = $devices[$i].qualified_name[$j].name $row | Add-Member -MemberType NoteProperty -Name "DNSFQDN" -Value $devices[$i].qualified_name[$j].name $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value $devices[$i].qualified_name[$j].name if ($currDevice.is_virtual -eq "false") { $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSFQDN" -Value $devices[$i].qualified_name[$j].name } } elseif ($devices[$i].qualified_name[$j].name_type -eq "FQDN") { $row | Add-Member -MemberType NoteProperty -Name "PhysicalDNSHostname" -Value $null $row | Add-Member -MemberType NoteProperty -Name "FQDN" -Value $devices[$i].qualified_name[$j].name } $j = $j + 1 } } } # Locations is an array containing one or more locations. We can store locations[0].name as the location $row | Add-Member -MemberType NoteProperty -Name "Location" -Value $devices[$i].locations[0].name # The Last Scan Date should always be there, but if absent PowerShell will happily record a NULL value for us. $row | Add-Member -MemberType NoteProperty -Name "LastScanDate" -Value $devices[$i].last_scan # Get extra data about the host $currDevice = Invoke-RestMethod $devices[$i].self -Credential $credential $row | Add-Member -MemberType NoteProperty -Name "OS" -Value $currDevice.operating_system.name if (!$currDevice.os_install_date) { $row | Add-Member -MemberType NoteProperty -Name "OS-Install-Date" -Value "(not available)" } else { $row | Add-Member -MemberType NoteProperty -Name "OS-Install-Date" -Value $currDevice.os_install_date } if (!$currDevice.bios.name) { $row | Add-Member -MemberType NoteProperty -Name "Bios" -Value $null } else { $row | Add-Member -MemberType NoteProperty -Name "Bios" -Value $currDevice.bios.name } # IP Address is an array of 0 or more IP Addresses. We might not have any, or we might have LOTS. $num = $currDevice.ip_address.count if ($num -eq 0) { $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value "(no IP address)" } else { if ($num -eq 1) { $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value $currDevice.ip_address[0].ip_address } else { # list of addresses, needs to be separated by semi-colons $addresslist = "" $j = 0 while ($j -lt $num) { $addresslist = -join($addresslist, $currDevice.ip_address[$j].ip_address) if ($j -lt $num) { $addresslist = -join($addresslist, ";") } $j = $j + 1 } $row | Add-Member -MemberType NoteProperty -Name "IP-Address" -Value $addresslist } } # Is this device physical or virtual if ($currDevice.is_virtual -eq "false") { # This is a physical target # We can fill in physical hardware details directly, and leave the entries for virtual hardware blank # Physical device RAM $row | Add-Member -MemberType NoteProperty -Name "PhysicalRAM" -Value $currDevice.total_memory_mb # Physical Device CPU Info # Sometimes we don't get CPU Information if ( (!$currDevice.cpu_count) -or ($currDevice.cpu_count -eq 0) ) { # Physical CPU info not avialble $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" } else { # we have at least some CPU info $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value $currDevice.cpu_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value $currDevice.core_count # Further details will be derived from cpu[0] if present if (!$currDevice.cpu[0].manufacturer) { # additional info missing $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" } else { $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value $currDevice.cpu[0].core_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value $currDevice.cpu[0].manufacturer $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value $currDevice.cpu[0].cpu_model $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value $currDevice.cpu[0].cpu_speed } } $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelSocketCount" -Value "n/a" # not exposed in V4 RestAPI $row | Add-Member -MemberType NoteProperty -Name "PhysicalModelCoreCount" -Value "n/a" # not exposed in V4 RestAPI $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceManufacturer" -Value $currDevice.manufacturer $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value $devices[$i].host_name # # Add this later # $row | Add-Member -MemberType NoteProperty -Name "PhysicalMACAddress" -Value $null } else { # This is a virtual target if (($currDevice.cpu_count -eq 0) -or (!$currDevice.cpu_count)) { # Physical CPU info not avialble $row | Add-Member -MemberType NoteProperty -Name "VirtualCPUCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "VirtualCoreCount" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUManufacturer" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" } else { $row | Add-Member -MemberType NoteProperty -Name "VirtualCPUCount" -Value $currDevice.cpu_count $row | Add-Member -MemberType NoteProperty -Name "VirtualCoreCount" -Value $currDevice.core_count } $row | Add-Member -MemberType NoteProperty -Name "VirtualRAM" -Value $currDevice.total_memory_mb $row | Add-Member -MemberType NoteProperty -Name "DeviceModel" -Value $currDevice.model # Do something with MAC addresses for VMs # Now, if the VM host we're running on was scanned, we can do something with the following: if (!$currDevice.virtual_host.device_id) { # host not scanned, record an error by putting "n/a" in physical hostname, leave everything else blank $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value "n/a" } else { # host was scanned, so let's populate as many details as possible about the physical host $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUCount" -Value $currDevice.virtual_host.cpu_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoreCount" -Value $currDevice.virtual_host.core_count if ( ($currDevice.virtual_host.cpu_count -ne 0) -and ($currDevice.virtual_host.core_count -ne 0) ) { $cpcpu = $currDevice.virtual_host.core_count / $currDevice.virtual_host.cpu_count $row | Add-Member -MemberType NoteProperty -Name "PhysicalCoresPerCPU" -Value $cpcpu } # In order to get the CPU Manufacturer information, you need to follow the link to # $currDevice.virtual_host.self and parse the CPU info there. This is left as an exercise to the user $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUModel" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalCPUSpeed" -Value "n/a" $row | Add-Member -MemberType NoteProperty -Name "PhysicalRAM" -Value $currDevice.virtual_host.total_memory_mb $row | Add-Member -MemberType NoteProperty -Name "PhysicalDeviceManufacturer" -Value $currDevice.virtual_host.manufacturer $row | Add-Member -MemberType NoteProperty -Name "PhysicalHostname" -Value $currDevice.virtual_host.host_name # In order to get the MAC Address for the physical host, you need to follow the link to # $currDevice.virtual_host.self and parse the MAC info there. This is left as an exercise to the user $row | Add-Member -MemberType NoteProperty -Name "PhysicalMACAddress" -Value "n/a" } } $csv += $row $i = $i + 1; # keep track for inner loop $seen = $seen + 1; # keep track for outer loop if ( $seen % 10 -eq 0) { # progress indicator - display a "." every 10 devices write-host "." -nonewline } } # Finished this batch $offset = $seen } # Save the output to a file write-host " Done. Saving output to OutputDevices.csv now." $csv | Select-Object Hostname, FQDN, OS, OS-Install-Date, Location, Serial-Number, PhysicalCPUCount, PhysicalCoreCount, PhysicalCoresPerCPU, ` PhysicalCPUManufacturer, PhysicalCPUModel, PhysicalCPUSpeed, PhysicalRAM, VirtualCPUCount, VirtualCoreCount, VirtualRAM, DeviceModel, ` PhysicalModelSocketCount, PhysicalModelCoreCount, PhysicalDeviceManufacturer, PhysicalHostname, PhysicalFQDN, IP-Address, ` PhysicalMACAddress, VirtualMACAddress, ClusterInformation, ClusterName, PartitioningMethod, DerivedCPU, DerivedCoresPerCPU, ` Bios, LastScanDate, DeviceID, PhysicalDeviceID, PhysicalCPUNotes, Notes, ExternalLink, DNSHostname, DNSFQDN, PhysicalDNSHostname, ` PhysicalDNSFQDN, MeasurementComment | Export-csv OutputDevices.csv -NoTypeInformation
Related articles