NOP, just NOP

Networking

Networking | 2011-06-23 11:59:16

Last night we upgraded one of our CallManager 7 clusters to a new minor version to resolve issues and to apply the usual bug fixes. We were moving from 7.1.3.33024-1 to 7.1.5.30000-1, which in the past we had not noticed any issues with.

In preparation, we preloaded 7.1.5.30000-1 about a week before the upgrade was scheduled to take place to reduce time in the outage window. After the upgrade had completed, we noticed that we were missing data, or that data had returned to previous values. The most obvious changes were to route patterns mobile connect setups (due to end user visibility), however other data loss may have also occurred.

We looked at the system and realised that our missing or modified data was any changes that had occurred in the week that the upgrade had been preloaded. It seemed that a database snapshot had been taken when we preloaded  7.1.5.30000-1 and that any changes made to 7.1.3.33024-1 had been ignored once the upgrade was applied.

We trolled through the install logs but could not find any errors during the upgrade or reason as to why this occurred.

The Cisco site doesn’t have any upgrade documentation for 7.1, but a similar doc from 6.1(1) states: Your configuration information migrates automatically to the upgraded version in the active partition. http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/cucos/6_1_1/cucos/iptpch7.html#wp1040186

Needless to say we will not be preloading future updates and that everything will be performed inside of the outage window to minimise data loss.



Networking | 2010-11-03 11:47:25

What happens if you want to communicate with a network that you don’t have a route for? In a normal situation you would just install a new route across the network, either statically or via a dynamic protocol like EIGRP or OSPF.

However what do you do if this isn’t an option? I had this problem the other day because the remote network was owned by another company and we didn’t want to request for routes to be installed or to run a routing protocol. However there was a mutual network that could access both ends. We decided to run up a NAT scenario, however for end to end communication to be successful we need to NAT both the source and destination addresses in each packet.
The diagram below should show things a bit better. In our situation we had Call Manager servers we needed to access, but the 10.10 network couldn’t hit the 10.20 network. So we NATted up on the 10.15 network.

Here’s how we achieved the NATting.

ip nat pool VOICE1 10.15.0.1 10.15.0.1 netmask 255.255.255.0
ip nat inside source list NAT_ACL_VOICE1 pool VOICE1 overload
ip nat pool VOICE2 10.15.0.2 10.15.0.2 netmask 255.255.255.0
ip nat inside source list NAT_ACL_VOICE2 pool VOICE2 overload
ip nat pool VOICE3 10.15.0.3 10.15.0.3 netmask 255.255.255.0
ip nat inside source list NAT_ACL_VOICE3 pool VOICE3 overload
ip nat outside source static 10.20.0.1 10.15.0.1
ip nat outside source static 10.20.0.2 10.15.0.2
ip nat outside source static 10.20.0.3 10.15.0.3
ip route 10.15.0.1 255.255.255.255 10.20.0.1
ip route 10.15.0.2 255.255.255.255 10.20.0.2
ip route 10.15.0.3 255.255.255.255 10.20.0.3
ip access-list extended NAT_ACL_VOICE1
remark PERMIT LOCAL NETWORK
permit ip 10.10.0.0 0.0.0.255 host 10.15.0.1
deny ip any any
ip access-list extended NAT_ACL_VOICE2
remark PERMIT LOCAL NETWORK
permit ip 10.10.0.0 0.0.0.255 host 10.15.0.2
deny ip any any
ip access-list extended NAT_ACL_VOICE3
remark PERMIT LOCAL NETWORK
permit ip 10.10.0.0 0.0.0.255 host 10.15.0.3
deny ip any any

A quick run down on how I *think* it’s working, correct me if I’m wrong.

To change the source addressing information we create a NAT pool with one valid IP (our NATted IP). To filter this from NATting all traffic we apply a NAT ACL onto the pool and overload it (PAT).
To change the destination addressing information we create an outside static NAT.
Because of the way outside NATting works, it will route the packet before NATting it. Eventually one of our routes will match the remote network but this won’t apply the NAT properly. To get around this we force a longest match route, which will always match and NAT properly.

Finally we just need to activate our interfaces for NATting.

int fa0/1
ip nat inside
int fa0/0
ip nat outside

This may not be the cleanest way to do this (and it’s pretty tedious making a new pool for every address you need) and I may have got the actual details about what is happening where not 100%, but this example is running in production now on one of my networks without any dramas.



Networking | 2010-10-20 23:38:24

Who would ever have thought I would have more issues with the bloody SIP trunk between CME and iiNet?

My CME has been offline for a good 6 months because I had unrelated IOS issues and got lazy fixing the proper config.

Today I dusted off my CME config for SCCP with a SIP trunk and got everything running smoothly. Inbound calls worked perfectly, 2 way audio, happy days. However outbound calls weren’t working. The phone would simply sit trying to call, no ringing tone, but after a minute or two I would get something similar to a fast busy.

After debugging with debug ccsip messages I soon found the reason why outbound wasn’t working. The way the SIP registration process works is inbound calls are determined by CME’s registration status. If you issue a show sip-ua reg stat it will show you if the SIP trunk is registered or not. If it is, great, inbound *should* work. However outbound is slightly different. Upon each call CME issues a INVITE which should acknowledged by a 100 Trying from the SIP provider. The provider will then issue SIP/2.0 401 Unauthorized as a challenge to authenticate in which you must ACK and INVITE again. This second invite will contain the response to the challenge. After you send the INVITE there should be another 100 Trying and then something like 183 Session Progress to say that everything is now working.

My problem was that after my INVITE with the challenge response I would never receive Trying, so CME would spam INVITE over and over again and then eventually time out.

In the middle of the ccsip messages debug is info about RTP and codecs. As it’s running over WAN I’m running G729 and had originally set the codec as g729br8. br8 is an annex B edition with built in VAD (which doesn’t sound too bad). In the debug i noticed: a=fmtp:18 annexb=yes. I figured by stating it there that it might have supported it, but I decided to try g729r8 anyway. As soon as I did this I started receiving the proper 100 and 183. Rock and roll.

SO YOU WOULD THINK IT WAS ALL OVER RIGHT?!?

No.

I then tested to make sure everything was good, inbound and outbound call flows. Now outbound was good, but whenever I tried calling in the call would not ring and hit straight through to iiNet’s voicemail. So I did another debug and now it showed: SIP/2.0 488 Not Acceptable Media. That’s weird right, because I was just using r8 as outbound? No worries, so I tried adding br8 as the second preference codec. Now when I tried to call out I received a message from iiNet saying that “I could not dial out at this time”

So I swapped things back and forwarded a few times and the conclusion I came to was I could only make outbound calls using r8 and receive inbound calls using br8. After messing with dial peers for most of the night with intermittent success I got frustrated with it and just changed everything over to g711alaw. No love on the oubound calls again. g711ulaw however seemd to work perfectly in all scenarios.

I’m so over getting this to work that I can’t be bothered at this stage to find out why 729 would only work in certain directions or when alaw doesn’t when in both situations iiNet say they support both. Maybe something for another day.



Networking | 2010-10-13 16:03:07

This is the list of IOS hardening suggested by Cisco as per the CCNP2 curriculum. This combined with the common security ACL in the last post should be a good basis for keeping the network boundary tight.

BootP
Default: enabled
Description: This service permits the router to act as a BOOTP server for ther network devices. Such a service is rarely needed in modern networks, and should be disabled.

(config) no ip bootp server

CDP
Default: enabled
Description: CDP periodically advertises information between Cisco devices, such as the type of device and Cisco IOS version. Such information could be used to determine vulnerabilites and launch specific attachsl. Unless needed inside the network, this service should be disabled globally or disabled on unneccessary interfaces.

(config) no cdp run
(config-if) no cdp enable

Configuration auto-loading
Default: Enabled (globally and interfaces)
Description: This service permits a router to automatically load a configuration file from a network server upon boot. This service should remain disabled when not needed

(config) no service config

FTP Server
Default: Disabled
Description: This service permits the router to act as an FTP server for specific files in flash memory. It should remain disabled when not needed.

(config) no ftp-server enable

TFTP Server
Default: Disabled
Description: This service permits the router to act as a TFTP server for specific files in flash memory. It should remain disabled when not in use.

(config) no tftp-server file-sys:image-name

NTP service
Default: Disabled
Description: This service both receives a time-of-day clock from an NTP server and allows the router to act as an NTP server to NTP clients. Correct time is necessary for accurate time stamps when logging messages. This service should be disabled if not needed, or restricted to only devices that require NTP services.

(config) no ntp server ip-address

Packet assembler/disassembler (PAD) service
Default: enabled
Description: This service allows access to X.25 PAD commands in an X.25 network. Such a service is rarely needed in modern networks and should be disabled

(config) no service pad

TCP and UDP minor services
Default: Enabled before 11.3 disabled after 11.3
Description: These services execute small servers (daemons) in the router, typically used for diagnostics. They are rarely used and should be disabled.

(config) no service tcp-small-servers
(config) no service udp-small-servers

Maintenance Operation Protocol (MOP) service
Default: Enabled (most ethernet interfaces)
Description: This service is a Digital Equipment Corporation (DEC) maintenance protocol. Such a service is rarely needed in modern networks and should be disabled.

(config-if) no mop enable

Simple Network Management Protocol (SNMP)
Default: Enabled
Description: This service permits the router to respond to queries and configuration requests. If not used, this service should be disabled. If needed, restrict access to the router via access controls lists (ACL) and use SNMPv3 for additional security features.

(config) no snmp-server enable

HTTP Configuration and Monitoring
Default: Device dependent
Description: This service allows the router to be monitored and configured from a web browser. SDM uses secure HTTP (HTTPS). If not used, this service should be disabled. If needed, restrict access to the router via ACLs and use HTTPS for encrypted data transfer.

(config) no ip http server
(config) no ip http secure-server

Domain Name Service (DNS)
Default: Enabled (client services)
Description: Cisco routers use 255.255.255.255 as the default address to reach a DNS server for name resolution. If not used, this service should be disabled. If needed, explicitly set the address of the DNS server.

(config) no ip domain-lookup

ICMP Redirects
Default: Enabled
Description: This service causes the router to send an ICMP redirect message when a packet is forwarded out the interface it arrived on. An attacker can use such information to redirect packets to an untrusted device. This service should be disabled when not needed.

(config) no ip icmp redirect
(config-if) no ip redirects

IP Source Routing
Default: Enabled
Description: This service allows the sender to control the route that a packet travels through a network. Such a service can permit an attacker to bypass the normal forwarding path and security mechanisms in a network. Because most network devices should not attempt to dictate their preferred path through the network, this service should be disabled.

(config)no ip source-route

Finger service
Default: Enabled
Description: The finger protocol (port 79) retrieves a list of users from a network device, which includes the line number, connection name, idle time and terminal location. Such information is also seen in the show users Cisco IOS command and can be used for reconnaissance attacks. This service should be disabled when not needed.

(config) no service finger

ICMP unreachable notification
Default: Enabled
Description: This service notifies a sender of invalid destination IP subnets or specific addresses. Such information can be used to map a network . This service should be disabled.

(config-if) no ip unreachables

ICMP mask reply
Default: Disabled
Description: This service sends the IP subnet mask when it is requested. Such information can be used to to map a network. This service should be disabled on interfaces to untrusted networks.

(config-if) no ip mask-reply

IP directed broadcasts
Default: Enabled (Enabled Cisco IOS prior to 12.0, disabled Cisco IOS later than 12.0)
Description: A directed broadcast can be used to probe or deny service to (via a DoS attack) an entire subnet. The directed broadcast packet is unicast until it reaches the router that is responsible for the segment. At that time, the packet becomes a broadcast for the specific segment. This service should be disabled.

(config-if) no ip directed-broadcast

IP identification service
Default: Enabled
Description: The identification protocol (RFC 1413) reports the identity of the TCP connection initiator. Such information can be used in reconnaissance attacks. This service should be disabled.

(config) no ip identd

TCP keepalives
Default: Disabled
Description: TCP keepalives help clean up TCP connections when a remote host has stopped processing TCP packets (such as after a reboot). This service should be enabled to help prevent certain DoS attacks.

(config) service tcp-keepalives-in
(config) service tcp-keepalives-out

Gratuitous ARP
Default: Enabled
Description: This service is the primary means used in ARP poisoning attacks. Unless needed, this service should be disabled.

(config) no ip arp gratuitous

Proxy ARP
Default: Enabled
Description: This service permits the router to resolve layer 2 addresses. This feature is only useful if the router is acting as a layer 2 bridge. Because this is unlikely in modern networks, this service should be disabled.

(config) no ip arp proxy


Networking | 2010-10-13 16:00:32

After browsing my CCNA Security books I noticed that it recommends blocking a large range of ports used for different services on the router and that are insecure on end devices. After compiling them all together, here is a working ACL that can be implemented.

Keep in mind to change the RFC1918 (implemented to conform with RFC2827) blocking depending on the topology and that this may block services that you want running.

remark DENY TCPMUX
deny tcp any any eq 1
deny udp any any eq 1
remark DENY ECHO
deny tcp any any eq 7
deny udp any any eq 7
remark DENY DISCARD
deny tcp any any eq 9
deny udp any any eq 9
remark DENY SYSTAT
deny tcp any any eq 11
remark DENY DAYTIME
deny tcp any any eq 13
deny udp any any eq 13
remark DENY NETSTAT
deny tcp any any eq 15
remark DENY CHARGEN
deny tcp any any eq 19
deny udp any any eq 19
remark DENY TIME
deny tcp any any eq 37
deny udp any any eq 37
remark DENY WHOIS
deny tcp any any eq 43
remark DENY BOOTP
deny udp any any eq 67
remark DENY TFTP-DC OK
deny udp any any eq 69
remark DENY FINGER
deny tcp any any eq 79
remark DENY SUPDUP
deny tcp any any eq 93
remark DENY SUNRPC
deny tcp any any eq 111
deny udp any any eq 111
remark DENY LOC-SRV
deny tcp any any eq 135
deny udp any any eq 135
remark DENY NB-NS
deny tcp any any eq 137
deny udp any any eq 137
remark DENY NB-DGN
deny tcp any any eq 138
deny udp any any eq 138
remark DENY NB-SSN
deny tcp any any eq 139
deny udp any any eq 139
remark DENY SNMP
deny tcp any any eq 161
deny udp any any eq 161
remark DENY SNMP TRAP
deny tcp any any eq 162
deny udp any any eq 162
remark DENY XDMCP
deny udp any any eq 177
remark DENY NETBIOS
deny tcp any any eq 445
remark DENY REXEC
deny tcp any any eq 512
remark DENY RLOGIN WHO
deny udp any any eq 513
remark DENY RSH RCP
deny tcp any any eq 514
remark DENY SYSLOG
deny udp any any eq 514
remark DENY LPR
deny tcp any any eq 515
remark DENY TALK
deny udp any any eq 517
remark DENY NTALK
deny udp any any eq 518
remark DENY UUCP
deny tcp any any eq 540
remark DENY NEW-WHO
deny tcp any any eq 550
deny udp any any eq 550
remark DENY IRC
deny tcp any any eq 667
remark DENY MS UPNP SSDP
deny tcp any any eq 1900
deny udp any any eq 1900
deny tcp any any eq 5000
deny udp any any eq 5000
remark DENY NFS
deny udp any any eq 2049
remark DENY XWINDOW
deny tcp any any range 6000 6063
remark DENY NETBUS
deny tcp any any range 12345 12346
remark DENY BACKORIFICE
deny tcp any any eq 31337
deny udp any any eq 31337
remark PERMIT NEEDED ICMP
permit icmp any any parameter-problem
permit icmp any any packet-too-big
permit icmp any any source-quench
remark DENY UNNEEDED ICMP
deny icmp any any
remark DENY UNROUTABLE ADDRESSES
deny ip any 0.0.0.0 0.255.255.255
deny ip any 10.0.0.0 0.255.255.255
deny ip any 127.0.0.0 0.255.255.255
deny ip any 172.16.0.0 0.0.15.255
deny ip any 192.168.0.0 0.0.255.255
deny ip any 224.0.0.0 15.255.255.255


Networking | 2010-08-17 18:14:36

Today I had an issue provisioning 3 new AIR-LAP1142N-N-K9’s with the NME-AIR-WLC6-K9 module.

The AP’s and controller were all setup ok, but for some reason the AP’s didn’t want to register via CAPWAP. I had one previous working AP (which after the fix I dunno how it managed to work) that had registered. Via CDP it showed that the working AP was running C1140-K9W8-M while the busted AP’s were all running C1140-RCVK9W8-M. The recovery image is supposed to become active when the main image fails to load so I thought it may have been an issue I needed to console in to fix. Luckily it wasn’t.

All AP’s received DHCP on the right VLAN and everything else was looking fine (ACL’s etc) it was just the registration phase that didn’t work. After talking with some other engineers we found the issue was as simple as the option 43 on DHCP. It had previously been set to an ASCII option instead of HEX. Even though the Cisco docs said to set it as a HEX TLV we had it working previously on ASCII. After we hit it back to HEX all the AP’s registered without any problem.



Networking | 2009-10-06 09:59:29

So the Australian Daylight Savings kicked in over the weekend again.

The details which I posted here 12months ago works perfectly again. One extra thing to note is after applying the change to the Windows taskbar time, the 797X’s will change their time automatically, the other phones wont though.

In order to force them to update their time, you will need to reset the devices from the Date/Time group.

Hope this helps 🙂



Networking | 2009-04-10 09:59:55

Well, I’ve been having issues with my SIP registration from iiNet working within Cisco CME.

When doing a debug ccsip all, it appeared that I wasn’t receiveing a SIP INVITE, and that I would constantly throw out REGISTER’s but not hear anything back.

Yesterday I started thinking that maybe there was a something in my firewall ACL that was blocking the connection, but when I looked at it i couldn’t see anything wrong with it. I decided to add permit tcp any any eq 5060 just to make sure things were happening, and then I saw this response:

10 permit tcp any any eq 5060 (18 matches)

So things were happening but something still wasnt right.

I had spent most of the day looking over the config and trying different solutions around the net but nothing helped. This morning I decided to revisit the config and started with the ACL’s. Then I noticed this.

240 deny udp any any eq 1024 (128 matches)

I had borrowed an ACL from our work access layer switches, designed to filter out commonly used virus ports and this was one of the entries. It looked like the SIP response from iiNet was replying on udp port 1024 for the INVITE message which was of course blocked. As soon as I removed this registration went straight through and calls started routing.

Hope this helps saves the headaches that i had.



Networking | 2008-11-30 14:10:23

//EDIT
Fixed, see above posts.

//EDIT
it looks like something is wrong in this config. CME works perfectly however the SIP registration fails, for some reason I never receive a SIP INVITE. So be wary when using this config.

I have had my Cisco 2621XM working with Call Manager Express for some time now, and have had calls routing through iiNet’s SIP Servers, however recently I lost some configuration and had to rebuilt it again.

So I have decided to post it up here.

Note that this is only the SIP and CME configuration and heaps more is needed to actually run the router.

 

aaa authentication login LOCAL_AUTH local
aaa session-id common
!
ip dhcp pool p900
network 10.5.0.0 255.255.0.0
dns-server 203.0.178.191
default-router 10.5.0.1
option 150 ip 10.5.0.1
domain-name cme
!
voice service voip
sip
localhost dns:iinetphone.iinet.net.au
!
!
voice class codec 1
codec preference 1 g729br8
voice translation-rule 1
rule 1 /02XXXXXXXX/ /02XXXXXXXX/
rule 2 /XXXXXXXX/ /02XXXXXXXXX/
!
voice translation-rule 2
rule 1 /02XXXXXXXX/ /XXXXXXXX/
!
!
voice translation-profile Incoming_Number
translate called 2
!
voice translation-profile Outgoing_Number
translate calling 1
!
tftp-server flash:P00308000400.bin
tftp-server flash:P00308000400.loads
tftp-server flash:P00308000400.sb2
tftp-server flash:P00308000400.sbn
!
sccp ccm 10.5.0.1 identifier 1
!
sccp ccm group 1
associate ccm 1 priority 1
!
!
dial-peer voice 1 voip
description STD Calls
translation-profile incoming Incoming_Number
translation-profile outgoing Outgoing_Number
destination-pattern .T
voice-class codec 1
session protocol sipv2
session target dns:sip.nsw.iinet.net.au
dtmf-relay sip-notify rtp-nte
no vad
!
!
sip-ua
authentication username 02XXXXXXXX password XXXX realm iinetphone.iinet.net.au
no remote-party-id
retry invite 4
retry response 3
retry bye 2
retry cancel 2
retry register 5
timers register 300
mwi-server dns:sip.nsw.iinet.net.au expires 3600 port 5060 transport udp unsolicited
registrar dns:sip.nsw.iinet.net.au expires 3600
sip-server dns:sip.nsw.iinet.net.au
!
!
telephony-service
load 7960-7940 P00308000400
max-ephones 5
max-dn 5
ip source-address 10.5.0.1 port 2000
system message CCM4
time-format 24
date-format dd-mm-yy
voicemail 9999
mwi relay
max-conferences 4 gain -6
moh flash:music-on-hold.au
web admin system name XXXX XXXX
dn-webedit
time-webedit
transfer-system full-consult
transfer-pattern ....
directory entry 1 0001 name XXXX
create cnf-files version-stamp 7960 Oct 14 2008 07:28:46
!
!
ephone-dn  1  dual-line
number 02XXXXXXXX
label Main Phone
description Main Phone
name Main Phone
no huntstop
!
!
ephone  1
description Home 7940
mac-address 0011.93B6.CE9C
speed-dial 1 04XXXXXXXX label XXXX
type 7940
button  1:1


Networking | 2008-10-12 10:04:20

As part of my current job, I manage IP telephony for a building with about ~600 endpoints.

In .AU we go from +10GMT to +11GMT for daylight savings which come into effect the first Sunday of October. When we swapped over the times for daylight savings we noticed that some of the phones pull time from different sources. I searched for a definitive answer but couldnt find one in time.

Here is what we experienced. We run Cisco Call Manager 4.2(3)sr3a. The 7970 and 7975’s pull time directly from the windows taskbar time. So whatever time is showing on the taskbar is the time that will be shown on the 70 and 75’s. The 7906, 7940 and 7960 are a bit different. They seemed to pull the time from the CCM Date/Time group. The DT group is influenced by the Windows time, however the DTG will apply DST automatically.

This is what we experienced. If Windows was set at +10GMT and the DTG was also set at +10GMT (both in the sydney tz) and the time was 1PM, the 7X’s would show 1PM but all other phones would show 2PM. This is because the DT group thinks hey, Windows is +10GMT but I know were in DST so i will apply another 1 hour.

We were worried about placing either the Windows or CCM time in different TZ’s for logging reasons. If something was to happen down the track, the logs still need to reflect the right time.

I believe that these inconsistencies may have been caused by a number of factors inclusive of old firmware versions and windows patches not being applied. However due to this being a production network, upgrades on the fly are risky to do.

The fix here was to apply +10GMT(Syd) to Windows, set at the right time, and also +10GMT to CCM, but place it in the Brisbane TZ. This removed the errors as Brisbane does not participate in DST, but also kept both times at +10 to satisfy the logs.

Hope this helps some one.