Masergy MPLS

See also Masergy Contract Prior to 2010 WNA used an MPLS service from AT&T. The Masergy implementation took place in 2010. Below is an an explanantion of the upgrade process From the schematic below you can see that at the Chelmsford head-end there is a 6mb pipe burstable to 42mb priovided by T3 fiber connections that was installed in 2010. At the other sites T1's are in place. At the Polar warehouse there is a slightly different set up. Because of the location it was not possible to install a cost effective T1 in to a warehouse with only 3 users. Instead a 2mb virtual port was installed with a Cisco WRV210 to terminate the connection [1]

<embed_document>/wikireedia/masergy.pdf</embed_document>

INTRODUCTION AND BACKGROUND

AT&T has been providing network services to WNA since before 2003. In 2003 AT&T provided Frame Relay circuits. These were private line ‘always on’ circuits that ran between nodes (sites) on the network. Dedicated private line circuits are expensive and at the end of 2004, WNA took the opportunity to utilize the internet in order to carry network traffic. The AT&T Virtual Tunneling Service (AVTS) enabled WNA to move network traffic on an encrypted and secure tunnel on the public internet. This service could be provided at a fraction of the cost of the Frame Relay service and because the internet is everywhere we were confident of its ability to provide this service almost anywhere regardless of the location we wanted to connect to. The service was rolled out to all sites except the City of Industry Warehouse and, later, the Polar Warehouse because it was cost prohibitive. A work around was to use a local provider of network services and patch it in to the WNAGLOBAL network using DSL.

In 2007, WNA took a further step and upgraded the network to MPLS. The benefits of MPLS is that it is protocol agnostic (it can be deployed over the internet, Ethernet, etc) and the performance is much higher than equivalent protocols However, this implementation was not entirely successful. Each WNA site has two circuits. One circuit is for internal traffic and one is for internet traffic that is routed through the Chelmsford firewall. AT&T were able to upgrade the circuits for internal traffic but for reasons that AT&T were unable to explain, the cut over to MPLS for internet traffic did not work and AT&T kept WNA on the AVTS network for internet connectivity. What is more AT&T were not able to provide the MPLS service in Montreal immediately and Polar Plastic was kept on an AVTS circuit for both internet and internal network traffic. Despite the failure of the MPLS upgrade for internet traffic, AT&T started to charge for the service and what started was a two year fight to remove the failed circuits and stop the billing. By the middle of 2009 AT&T concluded that MPLS for internet traffic would never work given WNA’s configuration and announced that MPLS would never roll out MPLS to Montreal. By the end of 2009 AT&T had agreed a total of $140,000 in credits.

• Site Network Internet
• Chattanooga MPLS-T1 AVTS-T1
• Chelmsford MPLS-4xT1 AVTS-4xT1
• City of Industry office MPLS-T1 AVTS-T1
• City of Industry warehouse Verizon DSL with AVTS
• Covington MPLS-T1 AVTS-T1
• Lancaster MPLS-T1 AVTS-T1
• Polar office AVTS-T1 AVTS-T1
• Polar warehouse Local DSL with AVTS

Latterly AT&T proposed that we migrate to a newer MPLS-PNT service in the US and a slightly different version for the Montreal facility. At this point the cost of replacing the network was put out to quote to four additional vendors.

INITIATIVE TO REPLACE THE CURRENT AVTS/MPLS CIRCUITS

As part of a new initiative to replace the current network, WNA wants to make sure that in the future we can exploit new technologies as they become more mainstream. In the past year WNA has or is about to implement third party Cloud Computing applications (internet based) like Adaptive Planning, Syncada, Transite and Mimecast. In addition, WNA has implemented more applications that run over the network like the PLM (Product Lifecycle Management) and the internet-based Packaging Specification database and WNA may implement a WMS application. In future WNA will want to make use of increasingly mainstream technologies like VoIP, Video and Unified Messaging. As a whole WNA will require significantly more bandwidth to accommodate these applications as well as better tools to monitor the network and to easily increase or decrease network bandwidth as a whole or by application.

Over the past few years network security and compliance has become an increasing requirement from legislative and audit bodies. Consequently WNA not only needs more comprehensive security tools, but these tools needs to give more flexibility for the Network Administrator to make changes to security configurations as needed.

Another current problem WNA faces is the poor infrastructure of the ‘last mile’ providers. These are the carriers who are responsible for maintaining the local loop. All national and international internet and network traffic is carried by a single global network provider like AT&T or sometimes by multiple providers such as AT&T and CinBell partnering together. However the ‘local loop’ is owned by the local telecom provider in each location. They carry the ‘last mile’ of the service and are responsible for maintaining the physical infrastructure. As a company we have no contract with the local provider. Any contact that we do have is through our contracted network provider. Over the past five years we have experienced outages at City of Industry, Lancaster and more recently Chelmsford directly attributable to the actions of the local telecom company. As part of this initiative we want to ensure that even if the network provider cannot make the local telecom company upgrade their equipment, they will take responsibility on their SLA to compensate WNA for any outages on the network as well as put pressure on the local loop providers to upgrade their infrastructure.

CLASS OF SERVICE

The method by which networks allocate bandwidth is called Class of Service. (See appendix for definition of Class of Service) CoS is not something that we have considered in the past for our two main Enterprise applications (Email and ERP) but will be essential moving forward and will add to the overall cost. (In the quotes we received, CoS cost between $8 and $67 per month per site). This will make it far easy to prioritize network activity by the class of service that runs over it. That is data, video, voice, etc.

MANAGED OR NON-MANAGED NETWORK

One way WNA could reduce cost would be to manage the network ourselves. This would mean purchasing and maintaining our own routers, concentrators and firewall appliances at the Data Center and at the sites. This would save WNA money because buying the equipment is cheaper than leasing it on a monthly basis over three years or more. It is also likely that the bundled cost of the service would fall slightly to take in to account the absence of proactive monitoring and support. The network outside the CPE (Customer Premises Equipment) is still managed by the Network provider and they would still be responsible for any outage, latency, jitter and packet loss. Nonetheless, in the final analysis this approach was rejected in favor of a managed service in the hands of a technical team that would have more experience and tools at hand to resolve and escalate issues. The network providers were therefore asked to quote including managed services.

Each vendor was provided a short questionnaire to fill out explaining how their product would provide the service requirements as provided (as outlined above) or to give an alternative solution that meets the criteria. All quotes are based on a three year agreement.

PUTTING THE QUOTES IN CONTEXT

WHAT DOES WNA PAY TODAY?

The cost of network services has been falling slowly over the past two years. The elimination of the Maryland Heights office and the Cincinnati Bell Hosting Center has reduced the combined bills by $3,900/month to $10,631 / month. This does not include the cost of the DSL services and City of Industry and Polar warehouses at an approximate cost of $200/month combined.

THE NEW AT&T QUOTE

The new AT&T quote reduces the cost by 8%. The new service is MPLS-PNT (Private Network Transport) service combines the MPLS network with the AVTS capabilities by creating a virtual private network that segregates WNAGLOBAL network from all other internet traffic. Because PNT does not permit access to the internet a separate set of MIS Ports need to be purchased. This is not the case with the other competing quotes we received. It will be noticed that AVTS is still the only way to connect to Montreal and therefore required at the Chelmsford facility also.

 

AT&T Provided a second quote (Hi-Spec) that increases the bandwidth at Chelmsford from 3XT1 (4.5Mb) to 6Mb MLPPP. This would meet our future growth needs for internet traffic (Transite, Adaptive Planning, Syncada, etc)

ANALYSIS OF THE AT&T QUOTE

• AT&T does provide cheap and reliable remote user client software to securely connect to the network
• Increasing the bandwidth on internet access would be a good investment given our interfaces to third party applications for only a modest $340/month
• The AT&T quote still uses a combination of MPLS and AVTS networks for the City of Industry Warehouse and Montreal. This increases complexity running multiple products across the network although, it is no different than we do today. The quotes do not include the cost of providing network service to the warehouses or the Polar main office. I would estimate that would increase the total cost by $1,200 per month making it slightly more expensive than the current service at an estimated $11,250.
• The AT&T service does not provide the proactive monitoring tools that will allow us to see bandwidth usage by service/application nor is it possible to scale up or down the bandwidth at short notice to meet high demand at certain times of the year. (e.g. At Year End).
• AT&T does not provide any installation costs in the quotes.
• SLA does not include any credit due to network outage, jitter, latency, packet loss that they attribute to the local loop provider.
• AT&T did not quote for CoS on the network, This will increase the cost further.

 

QUOTES FROM ALTERNATIVE NETWORK PROVIDERS

MASERGY - RECOMMENDED

Masergy provided two quotes. Like AT&T, they were asked to quote on a like for like basis based on our current configuration and one that represents our future requirements. The first quote which is based on the same specification as AT&T and is comparable to our current service is quoted at $6,772 or about 30% lower.

The second Hi-Spec quote is significantly higher as first look. Their recommendation increases bandwidth to 10mb Ethernet at Chelmsford compared to 6Mb on the AT&T quote. (Equivalent to 6.6 T1’s). Ethernet is different from T1’s. In effect it is an extension of the Ethernet LAN that runs inside a building such as Covington and Chelmsford. As such it means that it is more robust and quicker that T1’s and is preferred over T1’s and can handle high volumes of data and video and voice. It is slightly more expensive than equivalent T1’s but there are distance limitations and therefore it is currently only offered in Metro areas such as Boston.

ANALYSIS OF THE MASERGY QUOTE

• The Masergy Hi-Spec proposal is 5% cheaper than the adjusted AT&T proposal.
• It includes the cost CoS unlike the AT&T quote.
• Masergy take responsibility for the whole network up to the CPE. That includes the local loop. Therefore, if WNA experience outages, jitter, latency, packet loss that falls below the 99.99% SLA, WNA is eligible for a credit irrespective of whether issue it is the fault of the local loop provider. Masergy say that they are the only network provider that covers the local loop and extensions.
• Masergy have included the cost of providing the service at the City of Industry and Polar Warehouses. If those sites remained on their DSL the quote would be reduced to approximately $9,000/month.
• The Masergy monitoring tools will enable WNA to determine by server (that is the ip address of the applications that are consuming the bandwidth. If necessary this can be defined down to an individual PC).
• Masergy will waive $14,441 of installation costs.
• Masergy will give WNA a two-month credit worth $21,415 if WNA signs an agreement by the end of the First Quarter 2010.
• Masergy have provided references for companies that are roughly the same size as WNA.

BACKGROUND ON MASERGY

Masergy are based in Plano Texas. There are privately-held company with revenues of $92 million (2008) with 200 employees. They operate in the America, Europe, Asia and Australia. They are a relatively small outfit compared to AT&T, Qwest, Verizon and even Cincinnati Bell. Masergy have provided evidence of financial stability and growth, Dunn and Bradstreet ratings and other financial data, which is attached is separate document.

QWEST

The Qwest quote comes in slightly cheaper than the AT&T and Masergy quotes on the comparison to our current costs. There would be a $1,536 installation charge at Polar.

• Qwest has a number of remote worker options with varying tariffs costing from about $20 to $50 per user per month in the US
• The service includes CoS prioritization.
• Qwest provides some good proactive monitoring tools.
• They quoted Polar and City of Industry sites for DSL but they did not indicate any access costs for connection to the their network via third-party DSL providers.
• They say that they are responsible for the circuit end to end and will work with the local loop provider to solve problems and supply a monitoring tool. Their SLA and repeated responses from Qwest fail to categorically say that WNA would receive rebates for Verizon equipment failure.
• No installation charges were quoted for the US sites but it is very likely that there would be installation charges
• Qwest did not quote a Hi-Spec network.

ANALYSIS OF THE QWEST QUOTE

The Qwest quote fairs favorably with AT&T but in itself would not be enough of an improvement to replace AT&T. They are ambiguous as to whether their SLA extends to the CPE. If it does not WNA would still be without credits when the local loop fails and, like AT&T, have less incentive to force Verizon to upgrade their infrastructure if they are not financially penalized. Qwest have had financial and customer support problems in the past but anecdotal evidence suggests that this has improved in the past few years.  

CINCINNATI BELL

The Cincinnati Bell Like-for-Like quote was more expensive than the Masergy and Qwest quotes.

• They quoted Quality of Service on the circuits.
• They did not provide a ‘Hi-Spec’ quote.
• CinBell have not quoted for the City of Industry or Polar warehouses or to get access to the network via the locally sourced network provides. This would increase the quote further.
• CinBell did not quote remote access client licenses.
• The CinBell SLA does not include outages emanating from local loop.

OTHER NETWORK PROVIDERS

Quotes were also sought from Verizon and One Communication. Verizon took 4 months to reply from initial contact. They failed to understand the network schematic or what we were looking for. They were continually chased for updates and finally responded with an inappropriate quote that has not been included here. One Communication withdrew from the process as they did not operate in Canada  

NEXT STEPS AND OTHER CONSIDERATIONS

The recommendation is to go with the Masergy Hi-Spec quote. If we were to accept their quote but continue to provide third-party access at City of Industry Polar warehouses the cost of their services would be approximately $9,000 per month. This is 8% cheaper than the current AT&T cost but includes a significantly upgraded network configuration. Prior to agreeing a contract with Masergy we would need to take references. We also need to determine their capacity to support WNA. If these questions are answered satisfactorily then proceed to drawing up an agreement.

When implementing a new network there is a cut over period during which WNA moves to the new network while maintaining the old one. There are a couple of reasons for this. One is an insurance against a major problem materializing with the new network. Secondly, a disconnection of an existing circuit takes 30 to 60 days and you would not take the decision to disconnect just in case there are delays in the new implementation. This will be true if we move to a new network provider or even if we stay with AT&T. Therefore regardless of the provider we will incur one to two months of double billing. However, if we proceed with Masergy, they are currently willing to provide us a two month credit to offset these charges. The implementation plan will be devised with the carrier including a signed off schematic and cut over plan at Chelmsford and the other sites. Typically these cut-overs will occur over night or during the weekend and will require both Masergy and IT personnel to connect the new networking gear to the newly installed circuits at the specified times. It is suggested that WNA plan to make the cut over 1 month in to fiscal 2011 (May 2010)

AT&T does not need to be involved with the cut over although they will be contractually obliged to support their infrastructure until the disconnections are complete.WNA IT has experience of two major network cutovers in the past five years and gained a lot of knowledge of what to expect and pitfalls to avoid. We are confident therefore that the cut over would be a well planned, low risk project.

Currently WNA still has a credit balance of $50,000 with AT&T and AP requested a refund check in January that will take one to two billing cycles to process. Nonetheless, AT&T as per the terms of resolution of dispute agreement, this credit is still owed to WNA and we are not bound to renew contacts.

If WNA were to cut over to a new Network in May the anticipated network costs are shown in the table below. WNA’s MPLS contract expires in September so WNA would be obliged to pay a further 5 months charges. Two of the months would be consumed for the cut over period where we continue to run in parallel in any case but we would be paying for 3 months MPLS charges after the service was disconnected. However by May we would still have a $40k credit against that charge. The AVTS MPLS contract has expired but we would continue to run in parallel for two months until the circuit is disconnected.

Masergy provide a two month credit at the start of the contractual period and there are no start up or implementation costs. It is estimated that in its first year the network would cost $98.5k net and $120k a year after that.

Note: AT&T still has the contacts to supply local and long distance voice networks at all the sites and a decision as to whether to renew those contacts will be made separately to this initiative although any impact on T1-PRI lines in Chelmsford will be determined.

We need to take additional steps to guarantee the success of the implementation by continuing to put pressure on Verizon to upgrade the local loop. We can do this by contacting local businesses surrounding WNA Chelmsford and leveraging their pressure alongside the Utilities board and the new network carrier to make the upgrade. WNA should also consider as a last resort making investment in fiber in full or agree to share the costs with Verizon.

Finally, in case we do have outages we need a redundant solution in emergencies. Therefore IT will investigate the purchase of mini mobile hotspots. This will enable internet access from the sites for a limited number of users at a very low monthly cost. For instance, a mini mobile hotspot would allow uses internet access to continue to work in Powertrack, Adaptive Planning, Transite, etc. 

APPENDIX

CLASS OF SERVICE

Class of Service is a 3bit field that that tags Ethernet frames that can be used by Quality of Service to differentiate traffic on a network. Traffic can include data, voice, video, etc. CoS specifies a value between 0 and 7 for each type of traffic with 7 being the highest priority. Traffic tagged at 7 would experience the highest quality of data transmission. This means it experiences the lowest level of Jitter, Latency, Packet loss. Typically the network provider would assign the highest quality of data to video data which has a low tolerance or Jitter and Latency and Packet loss and a lower number for slower streams of data. Network carriers will express their CoS in terms of bandwidth. In other words a particular QoS value would be guaranteed on a particular type of traffic up to 256k, 512k 756k, etc Priority Level Traffic Type

• 0 (lowest) Best Effort
• 1 Background
• 2 Standard (Spare)
• 3 Excellent Load (Business Critical)
• 4 Controlled Load (Streaming Multimedia)
• 5 Voice and Video

(Interactive Media and Voice)[Less than 100ms latency and jitter]

• 6 Layer 3 Network Control Reserved Traffic [Less than 10ms latency and jitter]
• 7 (highest) Layer 2 Network Control Reserved Traffic [Lowest latency and jitter]

LATENCY, JITTER, PACKET LOSS, ETC

Jitter – Means an inconsistent sending of packets of data. Rather than sending packets of data spaces out equally to the receiving node , they can get bunched up. On a streaming video transmission this would look like frozen frames followed by speeded up frames Latency – Packets of data take a long time to reach their destination. You can experience latency on a phone call when there is a delay on the line and there is a perceptible gap between one speaker’s question and the respondent’s answer. Packet Loss – Packets of data fail to reach their destination at all. You would experience packet loss on a streaming video transmission when the images on the picture jump from one part of the screen to the other rather than as a steady motion.

BRIEF DESCRIPTION ABOUT VLPS AND MPLS

Historically, since 1980s, Ethernet has evolved from a Local Area Network (LAN) protocol running at 10 Mbps over a shared coaxial cable to a latest networking technology running at speeds up to 10 Gbps over all manner of copper wire and optical fiber in networks extending worldwide. As the Ethernet popularity become larger, Service providers offer variety of carrier Ethernet Service to extend these network beyond the ‘LAN’ boundary into Wider access network environment, metropolitan area network(MAN).

As the network growth further, Multi Protocol Label Switching (MPLS) has become a great player for enterprise client to perform network interconnect between branches. MPLS Layer 2 point-to-point is an option for those whom require huge bandwidth for their network. Dedicated line will be physically planted at the clients dedicated places get link each other. However, the limitation of this giant trunk is that the service cannot go for point-to-multipoint services. Whilst, for those require lower bandwidth at their large number of sites, they would go for MPLS layer 3 IPVPN , which normally operates as a partitioned private network constructed over a shared IP-based backbone that utilizes technologies to ensure privacy of data.

Nonetheless, due to mega growth today in Data and video sharing, Service provider start to offer new service could comply on the huge multi-point transferred content. Virtual Private LAN Service (VPLS) would be a great player for such of those requirement. VPLS is a type of layer 2 VPN. In a VPLS, the Ethernet LAN at each customer site is extended as far as the edge of the provider network. The provider network then emulates the function of a LAN switch or bridge to connect all of the customer LANs to create a single bridged (Ethernet) LAN. One of the main differences between a VPLS and a MPLS point-to-point service is that the VPLS provides a point-to-multipoint service at layer 2.

VPLS has significant advantages for both service providers and customers. Service providers benefit because they can generate additional revenues by offering a new Ethernet service with flexible bandwidth and sophisticated service level agreements (SLAs). VPLS is also simpler and more cost effective to operate than a traditional service. Customers benefit because they can connect all of their sites to an Ethernet VPN that provides a secure, high speed and homogenous network. Moreover, VPLS provides a logical next step in the continuing evolution of Ethernet from a 10 Mbps shared LAN protocol to a multi-Gbps global service. Extracted from http://techviewz.org/2008/12/brief-description-about-vlps-and-mpls.html

RECENT CHANGES

• Although a Firewal (Fortinet) was installed at Covington, the intrenet traffic was still flowing through Chelmford. A gateway change was made so that users could divert their traffic through CVG on 192.168.20.1 This could be used as a alternative gateway for all users in the event of a disaster.
• Masergy interet certificate was renewed in early 2012

Ports

• Oracle/IFS
  • - 1521 Listener (added)
  • - 1585 Unknown (added) – already present as a listed Oracle port
  • - 62080 Solution Manager

• Radley EDI
  • - 8082 EDI Production

• Mattec
  • - TBD ProHelp
  • - TBD SQL

• Transite
  • - 5432 PostGreSQL

• Intek
  • - 1583 Pervasive (added)

• Crystal Reports
  • - TBD

• Citrix
  • - 1494