The Microsoft Azure landscape is changing drastically and it's doing a good job of moving resource management to a more modern view. Coupled with Microsoft's security initiatives (Intune, Defender, Sentinel, Copilots for Security), Azure ARC is a great way of managing on-prem servers for updates.

Microsoft has a few ways of enrolling on-prem machines into ARC, but it's tedious to do this without bulk enrolment. Currently they support Config Manager, Group Policy or Ansible bulk enrolment. There is a Powershell option, but I guess it's meant as a starting point for devs as it doesn't actually do much. Let's fix it and have it remote install via Powershell to domain joined machines.

Follow the initial steps of creating the subscription, resource group and service principal. Grab the latest "Basic Script" ie Powershell (as the below might be out of date) and wrap it around some Invoke-Command. Replace the values in <> that come from your script that it generates for you. 

# Read machine names from CSV file
$machineNames = Import-Csv -Path "arc_machines.csv" | Select-Object -ExpandProperty MachineName
$credential = (Get-Credential)

# Iterate through each machine
foreach ($machineName in $machineNames) {
    try {
        # Invoke-Command to run commands in an elevated context on the remote machine
        Write-Host "Attempting to install on $machineName"
        Invoke-Command -ComputerName $machineName -Credential $credential -ScriptBlock {
            # Code to execute on the remote machine
            $ServicePrincipalId="";
            $ServicePrincipalClientSecret="";
    
            $env:SUBSCRIPTION_ID = "";
            $env:RESOURCE_GROUP = "";
            $env:TENANT_ID = "";
            $env:LOCATION = "";
            $env:AUTH_TYPE = "principal";
            $env:CORRELATION_ID = "";
            $env:CLOUD = "AzureCloud";

            [Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor 3072;

            # Download the installation package
            Invoke-WebRequest -UseBasicParsing -Uri "https://aka.ms/azcmagent-windows" -TimeoutSec 30 -OutFile "$env:TEMP\install_windows_azcmagent.ps1";

            # Install the hybrid agent
            & "$env:TEMP\install_windows_azcmagent.ps1";
            if ($LASTEXITCODE -ne 0) { exit 1; }

            # Run connect command
            & "$env:ProgramW6432\AzureConnectedMachineAgent\azcmagent.exe" connect --service-principal-id "$ServicePrincipalId" --service-principal-secret "$ServicePrincipalClientSecret" --resource-group "$env:RESOURCE_GROUP" --tenant-id "$env:TENANT_ID" --location "$env:LOCATION" --subscription-id "$env:SUBSCRIPTION_ID" --cloud "$env:CLOUD" --correlation-id "$env:CORRELATION_ID";
        }
    }
    catch {
        Write-Host "Error occurred while connecting to $machineName : $_" -ForegroundColor Red
    }
}

Next create the file arc_machines.csv with one column called MachineName and each row being the DNS/NETBIOS name of the machine you want to remote into. The script will ask for your domain creds when starting which will be used to Invoke-Command into the remote host. It'll then use the Service Principal to enroll the machine into ARC.

Ubiquiti Unifi OS devices now restrict updates to its own platform when it thinks a disk is unhealthy. This is the most pointless and infuriating UX change the Ubiquiti has made (and they've made a few terrible changes in the past).

At-risk disks are flagged:

• Bad sectors being reported on the disk
• Uncorrectable disk errors detected
• Failures reported in the disk’s Self-Monitoring, Analysis and Reporting Technology system (SMART)
• The disk has reached 70% of the manufacturer’s recommended read and write lifespan (SSDs)

To not allow updates to their own platform stagnates feature development, bug fixes and security enhancements. Sure, make the admin aware that they have an at-risk disk, but don't prevent them from getting your latest updates, that's just purposefully punishing your customers over things that are not their fault.

In my case, my disk was old, but still healthy. I am not sure whether the power on hours were a flag from SMART or maybe the temperature went over some arbitrary threshold they decided to implement.

I was able to work around this by manually upgrading the platform.

• Enable SSH on the controller
• Use the ubnt-systool and associated FW links as documented here: https://help.ui.com/hc/en-us/articles/204910064-UniFi-Advanced-Updating-Techniques
• Reboot and enjoy

Sometimes even in the most organised of worlds, we still manage to miss patching older systems. I found an old HP iLO server running iLO 4 - 1.20 with no way to log into it. Every modern browser and OS has now deprecated old TLS, RC4 and 3DES cyphers for certificates with the most common FF error being thrown: SSL_ERROR_NO_CYPHER_OVERLAP.

Irrespective of what I tried (such as security.tls.version.min, version.fallback-limit, IE compatibility modes) nothing would work. Even an old 2008 server running IE wouldn't work because of Javascript blocking etc. Unfortunately the only way I could work around this was to spin a Win7 machine running original IE to bust into it. Ideally I wouldn't have had to roll a new Win7 VM, but old versions for this reason should be in everyones toolbox.

Once in, it's easy enough to upgrade to a more modern iLO FW which supports modern TLS considering that HP make it readily available https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_729b6d22f37f4f229dfccbc3a9.

This is the computed list of SSH bruteforce IP’s and commonly used usernames for April 2013.

Top 50 SSH bruteforce offenders IP’s.

Top 50 SSH bruteforce usernames.

As mentioned, I can confirm that Cisco Call Manager 9 (CCM9 ) does work in VirtualBox and can be installed in a similar manner to CCM7. I have had both 9.0.1 and 9.1.1 have been installed with all services running perfectly.

As we did with CCM7, CCM9 must first be installed in VMware and then moved over to VirtualBox. CCM9 is now 100% supported in VMware, so the install process should be flawless. Keep in mind though that VirtualBox is definitely not officially supported, so you will get no help from TAC. This should only be used in a lab environment.

The minimum requirements for CCM9 are the same as they were in CCM7, 1x 80GB SCSI disk with 2048MB RAM. The CUC prerequisites have changed slightly and if you use 80GB/2048MB you won’t be able to install CUC. I haven’t been bothered to find the minimum requirements for CUC but I’ll post them up when I get some time.

I’ve used VMware Workstation 8.0, but you should be able to use any version of VMware to build the initial machine. All we need to do is to have the install complete and boot successfully, all other finer details can be changed once we move over to VirtualBox.

1. Start by creating a new VM and choose a custom config.
2.  Depending on your version of VMware this may change, but I used Workstation 8.0 as the hardware platform.
3. We don’t want to use the auto deployment scripts and we will need to modify the hardware before boot, so just choose the ISO later.
4. Any version of Red Hat should work here, but I used 64-bit version of Enterprise 6.
5. Name it appropriately.
6. One processor is enough but if you’ve got more resources to throw at it, you may be able to do it here as long as you match the same in VirtualBox later.
7. Same goes for the RAM. The minimum requirements call for 2048MB but if you’ve got more, chuck it in.
8. I hate using NAT, but it’s probably useful for labs. In any case I’ve got bridged here, but we will redo this step later in the VBox config.
9. Make sure you use SCSI here. I haven’t tried SAS but it may work too.
10. Create a new HDD.
11. Make sure this is set to SCSI, it won’t work with IDE here.
12. I’ve got the minimum as 80GB here, but if you’ve got more throw it here.
13. This is where the vmdk is stored, make sure you take note of the location as we will need this file later to import into VBox.
14. Finish it up.
15. Edit your VM before powering it on, we’ve got a few things to do here.
16. Select the CD/DVD drive and browse for your ISO.
17. Select your ISO.
18. I’ve finished up here, but if you want you can remove the floppy, sound cards etc.
19. Power on the VMWare image.
20. The install process here is exactly the same as a typical CCM9 install, I’ve included it just for the sake of doing so.
21.  
22. Notice here that CUC isn’t available because our hardware config is too low speccd.
23.  
24. This will take quite a while.
25. Once the installation has finished, log in and shut it down.
26. Now it’s time to fire up VirtualBox.
27. Add a new Red Hat 64-bit guest.
28. Make sure your memory size is the same as what you built in VMware.
29. We need to not add a new hard drive here (we will be reusing the one built by VMware).
30. Just accept this.
31. We need to edit our VM before powering it on.
32. Remove the SATA controller, if you remember we built the VM in VMware using SCSI disks.
33. Add a SCSI controller.
34. Select Choose Existing Disk.
35. Browse to the vmdk file that was outputted by VMware.
36. Your disk setup should now look like this.
37. Choose the IDE CDROM drive to boot from the CentOS live boot disk. Note that you can boot of any live distro, I actually used the Ubuntu 12.04 live CD because I was having issues with remote key forwarding to the VM whilst using CentOS.
38. Again, I hate NAT’ed NIC’s so I switched mine to bridged.
39. Mount your CCM partition and chroot to it.
40. vi/nano/whatever the hardware_check.sh script in /usr/local/bin/base_scripts/ which is similar to what we did in CCM7.
41. Find the function check_deployment() as shown below.
42. Like we did for CCM7 edit out the isDeploymentValidForHardware function.
43. Make sure you save the file, I used vi to edit this so :wq! it.
44. Throw the following lines in to change the hardware type to match those by VMware.
    

    vboxmanage setextradata “<VM name>” “VBoxInternal/Devices/pcbios/0/Config/DmiBIOSVersion” “6 ”
    vboxmanage setextradata “<VM name>” “VBoxInternal/Devices/pcbios/0/Config/DmiSystemVendor” “VMware”
    vboxmanage setextradata “<VM name>” “VBoxInternal/Devices/pcbios/0/Config/DmiBIOSVendor” “Phoenix Technologies LTD”
    vboxmanage setextradata “<VM name>” “VBoxInternal/Devices/pcbios/0/Config/DmiSystemProduct” “VMware Virtual Platform”
    

45. Now you’re ready to fire up CCM9 in VirtualBox so just run that thang.
46. On bootup you should be able to see the OS detecting all your hardware as VMware devices – this is a good thing, don’t worry
47. If you receive some weird output, don’t worry too much, the important thing is that the OS boots and services start successfully.
48. Again, ignore any of these types of errors, this is why this shouldn’t be used in production.
49. Login, hooray!
50. Because the hardware has been modified slightly, the OS is unable to detect the vCPU and the amount of RAM.
    
51. However, everything still works perfectly 😉

Just a few notes about the install. In the CCM7 install I did before, I added a new user whilst chroot’ed over to the CCM partition so we could SSH in later to modify the check_deployment() script. I only attempted a few times, but every time I tried my SSH user couldn’t log in. All permissions were set correctly, the user was added to the OS properly but SSH wouldn’t work. I’m sure if I dug deeper I would probably find some sort of SSH permission script in Cisco’s funky land, but for the purposes of getting CCM9 into VirtualBox it wasn’t needed.

I’ll be posting some more info on the topic as I use this more. Also, due to CCM9’s new licensing model I *may* look at loading licenses on to get this running past the 60 day limitation.

Good luck

x90

Following on from the previous article I wrote about CCM7 in VirtualBox I can confirm that CCM9 can be installed in a similar manner. Both 9.0.1 and 9.1.1 have been installed with all services running perfectly.

I will post up a detailed guide on how to install and configure CCM9 in VirtualBox shortly.

I wanted to share some info on a recent issue I faced with Microsoft Hyper-V Server SP1 and trunking.

We are in an environment where we wanted to deploy a Hyper-V host to a site and have a trunk from a Cisco switch so that we could throw guests into different VLAN’s. Sounds relatively simple, and in the VMware environment it is, but Hyper-V proved a little more difficult to grasp.

Let me just run through a brief explanation of how the host is setup. We built the Hyper-V Server host, gave it an IP, added it to the domain and registered it in DNS. In SMVMM 2012 we defined a logical network in the fabric and added the VLAN and IP subnet (in CIDR) to the network site. We did not define an IP address pool for the logical network, as DHCP will be taking care of this for us. Next we added the host to VMM. We then deployed the host to the site and proceeded to modify the network settings to configure trunking remotely.

This was our downfall.

Remotely – in the host properties, we enabled logical network connectivity from out logical network, changed the NIC to trunk and made sure the subnet and VLAN details were correct. We then added the virtual network interface and granted host access through a VLAN so that we could still manage the host. Then we applied all the changed to the host.

What we did not know is that the host would apply settings one by one, and NOT send all config to the host to apply. What happened was that the host applied the trunking details and then could not apply the virtual network details as the host became offline.

After reading a tonne of unhelpful articles we solved the problem via the following:

• Cabling the second NIC.
• The second NIC got an IP via DHCP and hence moved the DNS record for the host.
• As the host was added to VMM using DNS, the network settings were applied properly which meant that the virtual network was created, bringing our trunked host back online.
• Once the host was up DNS had registered both IP’s so that we could safely shut down the second NIC, or allocate it to a management only NIC.

Hope this helps either explain a brief overview of trunking in Hyper-V or provide a solution to a similar problem.

Tonight I’ve been upgrading a CallManager 7.1.3 cluster to 7.1.5. This is a minor upgrade, but still follows upgrade procedures as usual.

I experienced two issues which I just need to note down.

Firstly, upon a switch version to the new partition CallManager takes A LONG TIME to start its services. The trap that I fell into was that I SSH’d to the box to check how the service start was going when I was greeted with this:

Requesting service status, please wait…
System SSH [STARTED]
Cluster Manager [STOPPED] Service Not Started
Service Manager [Not Running]
Warning: Service Manager need to be running for all Call Processing Applications

I started to freak out thinking that my new upgraded partition was fkd. I played with all sorts of starts and restarts shown below:

admin:utils service start Cluster Manager
Service Stopped
Cluster Manager [STOPPED]

admin:utils service start Service Manager
Starting servM
admin:

I rebooted the box hoping that something had got stuck and that it just needed a kick. When it came back up and SSH showed the same thing I tried to start services that I knew I hit the following:

admin:utils service start Cisco Database Layer Monitor
Service Manager [Not Running]
Warning: Service Manager need to be running for all Call Processing Applications
admin:utils service start Cisco Database Layer Monitor
Service Manager [Not Running]
Warning: Service Manager need to be running for all Call Processing Applications
admin:utils service start Cluster Manager
Service Stopped
Cluster Manager [STOPPED]

By this time I was freaking so I moved away for a few mins to work on something else. I came back and checked the service list again, and everything had started! So I guess the moral here is that 1. services don’t show as starting when they are starting for the first time. 2. an upgrade and switch partition takes a long time!

The second issue I had was a box not rebooting to its switched partition. I tried to force the switch but hit the following:

admin:utils system switch-version

Active Master Version: 7.1.3.32009-2
Inactive Master Version: 7.1.5.30000-1

If you are switching to an earlier release, you must run:
utils dbreplication reset all
from the publisher after all the nodes are switched over.
Do you really want to switch between versions ?

Enter (yes/no)? yes

Switching Version and Restarting the Appliance …

Switch version duration can vary depending on the database size
and platform configuration.  Please continue to monitor the
switchover process from here.

Waiting ………………………….

Operation failed

ERROR: Acquiring lock failed

I started to freak again. Then I found this page: http://iptbuzz.blogspot.com.au/2011/10/switch-version-doesnt-switch.html props to the guy who wrote that, because it worked perfectly! I manually rebooted the box and then forced a switch version and rebooted again and everything was gravy.

I’m yet to play with 8.x as we are waiting for 9 with proper VM support, but I hope that Cisco manage to fix their dodgey CCM scripting!

I thankfully received the extra 6 usb to serial cables today aswell as the Sunix USB PCI adapter. The GNS machine was rebooted, the drivers were loaded for the Sunix automatically and the additional usb to serial cables were registered without a hitch! For the first time since I started this quest I am able to console and telnet to all of my devices – I guess now the actual work starts.

Below are some new photos of the lab and telnet/console sessions to all of the devices.

More ranting and rambling bound to come now that config can begin.