Thursday, December 30, 2010

IP SLA

Cisco IOS IP SLAs can send SNMP traps that are triggered by events such as the following:

• Connection loss
• Timeout
• Round-trip time threshold
• Average jitter threshold
• One-way packet loss
• One-way jitter
• One-way mean opinion score (MOS)
• One-way latency

Alternately, an Cisco IOS IP SLAs threshold violation can trigger another Cisco IOS IP SLAs operation for further analysis.

The Cisco IOS IP SLAs MPLS VPN Awareness feature provides the capability to monitor IP service levels within Multiprotocol Label Switching (MPLS) Virtual Private Networks (VPNs). IP SLAs operations can be configured for a specific VPN by specifying a VPN routing and forwarding (VRF) name.

*************

The IP SLAs UDP jitter operation was primarily designed to diagnose network suitability for real-time traffic applications such as voice over IP (VoIP), video over IP, or real-time conferencing.

UDP jitter operations are capable of measuring the following:

Per-direction jitter (source to destination and destination to source)
Per-direction packet-loss
Per-direction delay (one-way delay)
Round-trip delay (average round-trip time)

Before configuring a UDP jitter operation on the source device, the IP SLAs Responder MUST be enabled on the target device (the operational target). The IP SLAs Responder is available only on Cisco IOS software-based devices.

When compared to the IP SLAs User Datagram Protocol (UDP) jitter operation, the IP SLAs ICMP jitter operation may provide less accurate measurements because the accuracy of the measurements provided by a non-Cisco destination device cannot be determined.

Since ICMP packets do not support voice technology, the IP SLAs ICMP jitter operation does not support Mean Opinion Score (MOS), Calculated Planning Impairment Factor (ICPIF), or estimated transmission rating factor (R) reaction configuration capabilities.

*************

The IP SLAs ICMP Jitter Operation feature provides the following key benefits:

End-to-end performance measurements between a Cisco device (source) and any other IP device (destination) using ICMP.

Proactive threshold violation monitoring through Simple Network Management Protocol (SNMP) trap notifications and syslog messages.

The IP SLAs ICMP jitter operation supports the following statistical measurements:

Jitter (source-to-destination and destination-to-source)
Latency (source-to-destination and destination-to-source)
Round-trip time latency
Packet loss
Successive packet loss
Out-of-sequence packets (source-to-destination, destination-to-source, and round-trip)
Late packets

*************

The IP SLAs UDP echo operation measures end-to-end response time between a Cisco router and devices using IP. UDP is a network layer (Layer 3) Internet protocol that is used for many IP services. UDP echo is used to measure response times and test end-to-end connectivity.

UDP echo accuracy is enhanced by using the IP SLAs Responder at Router A, the
destination Cisco router. If the destination router is a Cisco router, then IP SLAs sends a UDP datagram to any port number that you specified. Using the IP SLAs Responder is OPTIONAL for a UDP echo operation when using Cisco devices.

A UDP echo operation measures round-trip delay times and tests connectivity to Cisco and
non-Cisco devices.

*************

The IP SLAs HTTP operation measures the round-trip time (RTT) between a Cisco device and an HTTP server to retrieve a web page. The HTTP server response time measurements consist of three types:

DNS lookup—RTT taken to perform domain name lookup.
TCP Connect—RTT taken to perform a TCP connection to the HTTP server.
HTTP transaction time—RTT taken to send a request and get a response from the HTTP server. The operation retrieves only the home HTML page.

The DNS operation is performed first and the DNS RTT is measured. Once the domain name is found, a TCP Connect operation to the appropriate HTTP server is performed and the RTT for this operation is measured. The final operation is an HTTP request and the RTT to retrieve the home HTML page from the HTTP server is measured. One other measurement is made and called the time to first byte which measures the time from the start of the TCP Connect operation to the first HTML byte retrieved by the HTTP operation. The total HTTP RTT is a sum of the DNS RTT, the TCP Connect RTT, and the HTTP
RTT.

*************

The IP SLAs TCP Connect operation measures the response time taken to perform a TCP Connect operation between a Cisco router and devices using IP. TCP is a transport layer (Layer 4) Internet protocol that provides reliable full-duplex data transmission. The destination device can be any device using IP or an IP SLAs Responder.

*************

The IP SLAs ICMP Echo operation measures end-to-end response time between a Cisco router and any devices using IP. Response time is computed by measuring the time taken between sending an ICMP Echo request message to the destination and receiving an ICMP Echo reply. This operation does not require the IP SLAs Responder to be enabled.

*************

The IP SLAs ICMP Path Echo operation records statistics for each hop along the path that the IP SLAs operation takes to reach its destination. The ICMP Path Echo operation determines this hop-by-hop response time between a Cisco router and any IP device on the network by discovering the path using the traceroute facility.

The source IP SLAs device uses traceroute to discover the path to the destination IP device.

A ping is then used to measure the response time between the source IP SLAs device and each subsequent hop in the path to the destination IP device. This operation does NOT require the IP SLAs Responder to be enabled.

*************

The IP SLAs ICMP Path Jitter operation provides hop-by-hop jitter, packet loss, and delay measurement statistics in an IP network. The Path Jitter operation functions differently than the standard UDP Jitter operation, which provides total one-way data and total round-trip data.

The ICMP Path Jitter operation can be used a supplement to the standard UDP Jitter operation. For example, results from the UDP Jitter operation may indicate unexpected delays or high jitter values; the ICMP Path Jitter operation could then be used to troubleshoot the network path and determine if traffic is bottlenecking in a particular segment along the transmission path.

The operation first discovers the hop-by-hop IP route from the source to the destination using a traceroute utility, and then uses ICMP echoes to determine the response times, packet loss and approximate jitter values for each hop along the path. The jitter values obtained using the ICMP Path Jitter operation are approximates because ICMP only provides round trip times.

The ICMP Path Jitter operation is NOT supported in the RTTMON MIB; configuration and performance data can only be obtained using the CLI.

The ICMP Path Jitter operation functions by tracing the IP path from a source device to a specified destination device, then sending N number of Echo probes to each hop along the traced path, with a time interval of T milliseconds between each Echo probe. The operation as a whole is repeated at a frequency of once every F seconds. The attributes are user-configurable.

The IP SLAs ICMP Path Jitter operation is ICMP-based. ICMP-based operations can compensate for source processing delay but cannot compensate for target processing delay. For more robust monitoring and verifying, use of the IP SLAs UDP Jitter operation is recommended.

The jitter values obtained using the ICMP Path Jitter operation are approximates because ICMP does not provide the capability to embed processing times on routers in the packet. If the target router does not place ICMP packets as the highest priority, then the router will not respond properly. ICMP performance also can be affected by the configuration of priority queueing on the router and by ping response.

Unlike other IP SLAs operations, the ICMP Path Jitter operation is not supported in the RTTMON MIB. Path Jitter operations can only be configured using the CLI, and statistics can only be returned using CLI show ip sla commands.

In contrast with other IP SLAs operations, the IP SLAs Responder does NOT have to be enabled on either the target device or intermediate devices for Path Jitter operations. However, the operational efficiency may improve if you enable the IP SLAs Responder;

*************


The FTP operation measures the round-trip time (RTT) between a Cisco device and an FTP server to retrieve a file. FTP is an application protocol, part of the Transmission Control Protocol (TCP)/IP protocol stack, used for transferring files between network nodes.

Both active and passive FTP transfer modes are supported. The passive mode is enabled by default. Only the FTP GET (download) operation type is supported. The URL specified for the FTP GET operation must be in one of the following formats:

ftp://username:password@host/filename
ftp://host/filename

If the username and password are not specified, the defaults are anonymous and test, respectively. FTP carries a significant amount of data traffic and can affect the performance of your network. The results of an IP SLAs FTP operation to retrieve a large file can be used to determine the capacity of the network but retrieve large files with caution because the FTP operation will consume more bandwidth. The FTP operation also measures your FTP server performance levels by determining the RTT taken to retrieve a file.

*************

The DNS operation measures the difference between the time taken to send a DNS request and receive a reply. DNS is used in the Internet for translating names of network nodes into addresses. The IP SLAs DNS operation queries for an IP address if you specify a host name, or queries for a host name if you specify an IP address.

*************

The Dynamic Host Configuration Protocol (DHCP) operation measures the round-trip time (RTT) taken to discover a DHCP server and obtain a leased IP address from it. DHCP provides a mechanism for allocating IP addresses dynamically so that addresses can be reused when hosts no longer need them. IP SLAs releases the leased IP address after the operation.

There are two modes for the DHCP operation. By default, the DHCP operation sends discovery packets on every available IP interface on the router. If a specific server is configured on the router, using the ip dhcp-server command, discovery packets are sent only to that DHCP server.

The DHCP operation also measures your DHCP server performance levels by determining the RTT taken to obtain a leased IP address.

A DHCP relay agent is any host that forwards DHCP packets between clients and servers. Relay agents are used to forward requests and replies between clients and servers when they are not on the same physical subnet. Relay agent forwarding is distinct from the normal forwarding of an IP router, where IP packets are switched between networks somewhat transparently. Relay agents receive DHCP messages and then generate a new DHCP message to send out on another interface.

The IP SLAs DHCP operation contains a relay agent information option—Option 82—which is inserted by the DHCP relay agent when forwarding client-originated DHCP packets to a DHCP server. Servers recognizing the relay agent information option may use the information to implement IP address or other parameter assignment policies. The DHCP server echoes the option back verbatim to the relay agent in server-to-client replies, and the relay agent strips the option before forwarding the reply to the client. Option 82 includes three suboptions that convey information known by the relay agent:

circuit-id—identifies the incoming circuit.
remote-id—provides a trusted identifier for a remote high-speed modem.
subnet-mask—identifies the mask of the logical IP subnet from which the relay agent received the client DHCP packet.

This operation does NOT require the IP SLAs responder to be enabled so there
are no tasks to be performed on the destination device.


Prerequisites for Multioperation Scheduling of IP SLAs Operations

Configure the IP SLAs operations before group scheduling those operations.

Determine the IP SLAs operations you want to schedule as a single group.

Identify the network traffic type and the location of your network management station.

Identify the topology and the types of devices in your network.

Decide on the frequency of testing for each operation. Normal scheduling of IP SLAs operations allows you to schedule one operation at a time. If you have large networks with thousands of IP SLAs operations to monitor network performance, normal scheduling (scheduling each operation individually) will be inefficient and time-consuming.

The IP SLAs multiple operations scheduling functionality allows you to schedule multiple IP SLAs operations as a group using the ip sla group schedule command. The following parameters can be configured with this command:

Group operation number—Group configuration or group schedule number of the IP SLAs operation to be scheduled.

Operation ID numbers—A list of IP SLAs operation ID numbers in the scheduled operation group.

Schedule period—Amount of time for which the IP SLAs operation group is scheduled.

Ageout—Amount of time to keep the operation in memory when it is not actively collecting
information. By default, the operation remains in memory indefinitely.

Frequency—Amount of time after which each IP SLAs operation is restarted. When the frequency option is specified, it overwrites the operation frequency of all operations belonging to the group. Note that when the frequency option is not specified, the frequency for each operation is set to the value of the schedule period.

Life—Amount of time the operation actively collects information. The operation can be configured to run indefinitely. By default, the lifetime of an operation is one hour.

Start time—Time when the operation starts collecting information. You can specify an operation to start immediately or at an absolute start time using hours, minutes, seconds, day, and month. The IP SLAs multiple operations scheduling functionality schedules the maximum number of operations possible without aborting. However, this functionality skips those IP SLAs operations that are already running or those that are not configured and hence do not exist. The total number of operations will be calculated based on the number of operations specified in the command, irrespective of the number of operations that are missing or already running. The IP SLAs multiple operations scheduling functionality displays a message showing the number of active and missing operations. However, these messages are displayed only if you schedule operations that are not configured or are already running.

A main benefit for scheduling multiple IP SLAs operations is that the load on the network is reduced by distributing the operations equally over a scheduled period. This distribution helps you to achieve more consistent monitoring coverage. To illustrate this scenario, consider configuring 60 operations to start during the same 1-second interval over a 60-second schedule period.

If a network failure occurs 30 seconds after all 60 operations have started and the network is restored before the operations are due to start again (in another 30 seconds), then this failure would never be detected by any of the 60 operations. However, if the 60 operations are distributed equally at 1-second intervals over a 60-second schedule period, then some of the operations would detect the network failure. Conversely, if a network failure occurs when all 60 operations are active, then all 60 operations would fail, indicating that the failure is possibly more severe than it really is.

Operations of the same type and same frequency should be used for IP SLAs multiple operations scheduling. If you do not specify a frequency, the default frequency will be the same as that of the schedule period. The schedule period is the period of time in which all the specified operations should run.

The following sections explain the IP SLAs multiple operations scheduling process:

Multiple operations scheduling allows you to schedule multiple IP SLAs operations using a single command through the command line interface (CLI) or the CISCO-RTTMON-MIB.

This feature allows you to control the amount of IP SLAs monitoring traffic by scheduling the operations to run at evenly distributed times. You must specify the operation ID numbers to be scheduled and the time range over which all the IP SLAs operations should start. This feature automatically distributes the IP SLAs operations at equal intervals over a specified time frame. The spacing between the operations (start interval) is calculated and the operations are started. This distribution of IP SLAs operations helps minimize the CPU utilization and thereby enhances the scalability of the network.

The IP SLAs Multiple Operations Scheduling feature allows you to schedule multiple IP SLAs
operations as a group using the ip sla group schedule command.

The ip sla group schedule 1 1-10 schedule-period 20 [frequency 20] command is configured. This example schedules operation 1 to operation 10 within operation group 1. Operation group 1 has a schedule period of 20 seconds, which means that all operations in the group will be started at equal intervals within a 20-second period. By default, the frequency is set to the same value as the configured schedule period.

IP SLAs entry configuration commands:

  dhcp           DHCP Operation
  dns             DNS Query Operation
  ethernet      Ethernet Operations
  exit             Exit Operation Configuration
  frame-relay  Frame-relay Operation
  ftp              FTP Operation
  http             HTTP Operation
  icmp-echo    ICMP Echo Operation
  icmp-jitter    ICMP Jitter Operation
  mpls           MPLS Operation
  path-echo    Path Discovered ICMP Echo Operation
  path-jitter    Path Discovered ICMP Jitter Operation
  tcp-connect  TCP Connect Operation
  udp-echo     UDP Echo Operation
  udp-jitter     UDP Jitter Operation
  voip         Voice Over IP Operation


#ip sla reaction-configuration 1  react ?

 jitterAvg           Jitter Average in both the directions
 jitterDSAvg        Jitter Average in the direction from Destination to Source
 jitterSDAvg        Jitter Average in the direction from Source to Destination
                              
*************

#ip sla reaction-configuration 1  react jitterAvg threshold-type ?

  average       Average over N attempts
  consecutive  Consecutive occurrences
  immediate   React immediately
  never          Never react
  xOfy           X out of Y occurrences

*************

#ip sla reaction-configuration 1  react jitterAvg action-type ?
  none                No action
  trapAndTrigger  Trap and Trigger action
  trapOnly           Trap Only action
  triggerOnly       Trigger Only action

*************

  packetLateArrival          Packets arriving Late
  packetLoss                   Packet Loss in both direction
  packetLossDS               Packet Loss in the direction from Destination to Source
  packetLossSD               Packet Loss in the direction from Source to Destination
  packetMIA                   Missing In Action
  packetOutOfSequence   Packets arriving out of sequence

IP SLAs Reaction Configuration -

IP SLAs can be configured to react to certain measured network conditions. For example, if IP SLAs measures too much jitter on a connection, IP SLAs can generate a notification to a network management application, or trigger another IP SLAs operation to gather more data.

IP SLAs reaction configuration is performed using the ip sla reaction-configuration command. You can configure the ip sla reaction-configuration command multiple times so as to allow reactions for multiple monitored elements (for example, configuring thresholds for operation 1 for destination-to-source packet loss, and also configuring MOS thresholds for same operation). However, issuing the no ip sla reaction-configuration operation-number will clear all reactions for the specified operation. In other words, disabling of granular reaction elements (no ip sla reaction-configuration operation-number react monitored-element) is not currently supported, so as to provide backwards compatibility with the earlier version of this command.

You can check the configuration of the IP SLAs reaction configuration using the <show ip sla
reaction-configuration> command.

IP SLAs includes the capability for triggering SNMP notifications based on defined thresholds. This allows for proactive monitoring in an environment where IT departments can be alerted to potential network problems, rather than having to manually examine data.

IP SLAs supports threshold monitoring for performance parameters such as average jitter, unidirectional latency and bidirectional round trip time and connectivity. This proactive monitoring capability provides options for configuring reaction thresholds for important VoIP related parameters including unidirectional jitter, unidirectional packet loss, and unidirectional VoIP voice quality scoring (MOS scores).

IP SLAs can generate system logging (syslog) messages when a reaction condition occurs. These system logging messages can then be sent as SNMP notifications (traps) using the CISCO-RTTMON-MIB. For packet loss and jitter, notifications can be generated for violations in either direction (source to destination and destination to source) or for round trip values. Packet loss, jitter and MOS statistics are specific to IP SLAs Jitter operations. Notifications can also be triggered for other events, such as round-trip-time violations, for most IP SLAs monitoring operations.

SNMP notifications (traps) for IP SLAs can be configured as a triggered action, to be sent when monitored values exceed an upper threshold or fall below a lower threshold, or when a set of defined conditions are met. For example, an SNMP trap can be triggered by 5 consecutive timeouts during an IP SLAs operation.

The sending of SNMP traps is one of the options for triggered actions that can be configured for IP SLAs violations. The monitored values (also called monitored elements), the threshold type, and the triggered action are configured using the ip sla reaction-configuration global configuration mode command.

SNMP traps for IP SLAs are supported by the CISCO-RTTMON-MIB and CISCO-SYSLOG-MIB. Use the ip sla logging traps command to enable the generation of SNMP system logging messages specific to IP SLAs trap notifications. Use the snmp-server enable traps rtr command to enable the sending of IP SLAs SNMP trap notifications.

IP SLAs reactions are configured to be triggered when a monitored value exceeds or falls below a specified level, or when a monitored event (such as a timeout or connection loss) occurs. These monitored values and events are called monitored elements.

Round-trip-time (rtt) is one of the monitored values of all IP SLAs operations. Events (such as traps) can be triggered when the rtt value rises above a specified threshold, or when it falls below a specified threshold. To configure rtt as the monitored element, use the following version of the ip sla reaction-configuration command:

Jitter (interpacket delay variance) is one of the monitored values of IP SLAs UDP Jitter operations.

Jitter values are computed as source-to-destination, destination-to-source, and combined round-trip values. Events (such as traps) can be triggered when the average jitter value in either direction, or in both directions, rises above a specified threshold, or when it falls below a specified threshold.

Pactket loss is one of the monitored values of IP SLAs UDP Jitter operations. Jitter values are computed as source-to-destination and destination-to-source values. Events (such as traps) can be triggered when the jitter value in either direction rises above a specified threshold, or when it falls below a specified threshold.

To configure source-to-destination packet loss as the monitored element, use the react PacketLossSD syntax in the ip sla reaction-configuration command.

To configure destination-to-source jitter as the monitored element , use the react PacketLossDS syntax in the ip sla reaction-configuration command.

Mean opinion score (MOS) is one of the monitored values of IP SLAs Jitter VoIP operations. MOS values are computed as numbers to two decimal places, from a value of 1.00 (worst quality) to 5.00 (best quality). Events (such as traps) can be triggered when the MOS value in either direction rises above a specified threshold, or when it falls below a specified threshold.

To configure destination-to-source jitter as the monitored element , use the react mos syntax in the ip sla reaction-configuration command.

The threhold-type syntax of the ip sla reaction-configuration command defines the type of threshold violation (or combination of threshold violations) that will trigger an event. Threshold violation types are as follows:

immediate - Triggers an event immediately when the value for a reaction type (such as response time) exceeds the upper threshold value or falls below the lower threshold value, or when a timeout, connectionLoss, or verifyError event occurs.

consecutive - Triggers an event only after a violation occurs a specified number of times
consecutively. For example, the consecutive violation type could be used to configure an action to occur after a timeout occurs 5 times in a row, or when the round-trip-time exceeds the upper threshold value 5 times in a row

x of y - Triggers an event after some number (x) of violations within some other number (y) of probe operations (x of y)

averaged - Triggers an event when the averaged totals of a value for x number of probe operations exceeds the specied upper-threshold value, or falls below the lower-threshold value. Configuring these threshold violation types is described in the following sections.

Generating Events for Each Violation -

To generate a trap (or trigger another operation) each time a specified condition is met, use the immediate threshold-type keyword:

ip sla reaction-configuration operation-number react data-type threshold-type immediate
threshold-value raising-value falling-value action-type action-value

Generating Events for Consecutive Violations -

To generate a trap (or trigger another operation) after a certain number (x) of consecutive violations, use the consecutive keyword with the optional number-of-occurrences argument:
ip sla reaction-configuration operation-number react reaction-condition threshold-type consecutive [number-of-occurances] threshold-value raising-value falling-value action-type action-value. The default value for number-of-occurances is 5.

Generating Events for x of y Violations -

To generate a trap (or trigger another operation) after some number (x) of violations within some other number (y) of probe operations (x of y), use the xofy [x-value y-value] syntax:
ip sla reaction-configuration operation-number react reaction-condition threshold-type xofy x-value y-value threshold-value raising-value falling-value action-type action-value
The default x-value and y-value is 5 (xofy 5 5).

Generating Events for Averaged Violations -

To generate a trap (or trigger another operation) when the averaged totals of x number of probe operations violate a falling-threshold or rising-threshold, use the average [attempts] syntax:

ip sla reaction-configuration operation-number react reaction-condition threshold-type average [attempts] threshold-value raising-value falling-value action-type action-value
The default value for attempts is 5.

Specifying Reaction Events -

Action type options for the ip sla reaction-configuration command are as follows:

none—No action is taken.

trapOnly - Send an SNMP logging trap when the specified violation type occurs for the monitored element. IP SLAs logging traps are enabled using the ip sla logging traps command. For SNMP logging traps to be sent, SNMP logging must be enabled using the appropriate SNMP commands, including the snmp-server enable traps syslog command.

triggerOnly - Have one or more target operation's operational state make the transition from "pending" to "active" when the violation conditions are met. The target operations to be triggered are specified using the ip sla reaction-trigger command. A target operation will continue until its life expires, as specified by the target operation's configured lifetime value). A triggered target operation must finish its life before it can be triggered again.

trapAndTrigger - Trigger both an SNMP trap and start another IP SLAs operation when the violation conditions are met, as defined in the trapOnly and triggerOnly options above.

In the following example, IP SLAs operation 10 (a UDP Jitter operation) is configured to send an SNM logging trap when the MOS value exceeds 4.9 (best quality) of falls below 2.5 (poor quality)

# ip sla reaction-configuration 10 react mos threshold-type immediate threshold-value 490 250 action-type trapOnly

7 comments:

  1. Thanks! This is one of my more popular posts!

    ReplyDelete
  2. Hey AMB...
    This is a nice post and provides useful and concise information.
    I have a query which is listed below
    IP SLA Source--->R1---->R2----> R3 --- > R4
    Conside the above topology, where we have a ip sla source. is it possible to measue the response time on just the link between R2 and R3. If yes, pls speify how and if No, pls suggest an alternative

    Thanks

    ReplyDelete
  3. This should accomplish what you are looking for although you might need to parse the data a little.

    The IP SLAs ICMP Path Echo operation records statistics for each hop along the path that the IP SLAs operation takes to reach its destination. The ICMP Path Echo operation determines this hop-by-hop response time between a Cisco router and any IP device on the network by discovering the path using the traceroute facility.

    The source IP SLAs device uses traceroute to discover the path to the destination IP device.

    A ping is then used to measure the response time between the source IP SLAs device and each subsequent hop in the path to the destination IP device. This operation does NOT require the IP SLAs Responder to be enabled.

    ReplyDelete
  4. Hi AMB,

    As per this document we should be able to calculate the packet loss by configuring ip sla path jitter configuration without ip sla responder configuration.

    My requirement is: we have a internet link. Without any configuration at ISP end, i want snmp to sent notification when packet loss is 80%. Is this can be achieved? If so. plz guide on it.

    Thanks in advance!

    Regards
    Sunitha

    ReplyDelete
  5. Hey Sunitha,

    I am adding a link for you... What you are looking for is setting up a "threshold" and once that has been exceeded it will send a trap. That entire doc page is worth looking over.

    http://www.cisco.com/en/US/docs/ios/12_4/ip_sla/configuration/guide/hsthresh.html#wp1082288

    Router# ip sla monitor reaction-configuration 10 react jitterAvg threshold-type immediate threshold-value 5000 3000 action-type trapAndTrigger

    You might need to adjust the values but other than that, it should accomplish what you are looking to achieve.

    Good Luck!

    ReplyDelete
  6. There is definately a great deal to find out about this subject.
    I like all of the points you've made.

    my homepage ... heated cat house plans

    ReplyDelete