System of the Year: Cox Orange County/Palos Verdes

In 1999, Cox Communications consolidated seven Orange County, California, sites into one facility in Rancho Santa Margarita. It’s a massive, 300,000 square-foot building that spans more than 24 acres. Built for fifteen years earlier for another industry, Cox recast it for modern telecommunications purposes, and growth.

The year before, Orange County had become the first Cox system to launch residential cable telephony. Other systems followed, with incredible results. The track record was so impressive that, by the end of 2006, Telephony magazine topped a 3700-word article about Cox with this straight-faced headline: "The Best Phone Company in America?"

That trend continues, unabated. In the most recent J.D. Power & Associates Residential Regional Telephony Customer Satisfaction Survey, Cox placed first in the West Region, for the fifth year in a row. While each of the Cox Western (and, in cases, inter-related) systems contributed to that ranking, there are reasons to consider Orange County/Palos Verdes as a first among equals. There is telephony, but also the system’s role in Cox’s ongoing upgrade cycle, its delivery of business services, its search for better network data-mining techniques and its performance on a metric that increasingly bears watching, employee satisfaction.

And perhaps a needless reminder for anyone familiar with this company’s reputation for engineering and technical operations, but a leader at Cox is a leader in the industry at large.
Quiet voice storm One thing that strikes visitors to the Rancho Santa Margarita facility is its tone. "It’s a very, very quiet office," said one consultant who has made several treks. "An unusually professional environment."

The technical nerve center of Cox Orange County, however, is far from quiet. "I love to hear myself talking," apologized Rick Guerrero, OC/PV vice president of broadband engineering and operations, at the end of an interview. "No one else does!"

Guerrero’s voluble reputation indeed precedes him, but there are several reasons why someone might rather like – if not love – to hear what he has to say. He talks up the members of his team and vendor partners, too, who probably don’t mind the attention. He turns phrases well, and having been in charge of this engineering group since 1996, has many stories to tell.

Finally, Guerrero talks like a man of action. (After all, this is someone whose recreational activities of choice involve not meditating on a golf ball but driving Porches and peddling road bikes in foreign lands.) His words ultimately seem derivative of the energy it takes to accomplish numerous, big, complicated tasks.

Take that initial launch of constant bit rate (CBR) telephony. In retrospect, Cox’s reputation for thoughtful and deliberate corporate engineering may have clouded the reality of the risky confusion associated with those early days.

"It was a different time," he said. "We were partnering with anyone who knew anything about delivering upstream services to the level that you had to delivering telephone – vendors, systems, we were going to church regularly getting divine guidance every time we could."

The idea that Orange County helped write the book on cable telephony is accurate enough, with the additional element of severe time constraints. "I felt like I was in a choir, and the music director is writing music on the right, and I’m singing the page on the left," he said. "That’s how close it was."

The subsequent success that Cox overall has enjoyed in telephony is public record. As for how Orange County in particular has fared, it’s hard to say precisely. Operationally, the system began migrating its Nortel DMS 500 switches to handle voice over IP two years ago, and launched VoIP in Palos Verdes last year. A Cox spokesperson said that as of May this system had 280,000 basic cable subscribers, but declined to share data or telephone numbers.

It’s telling to note, however, that where this system – or Cox Omaha, for that matter – is concerned, the word "incumbent" no longer appears relevant. Internally, one also hears counterparts at Cox sister systems gently begrudging Orange County for its good demographics, implying that its superior telephony numbers are attributable, at least in part, to good territory.

The bottom line is that this system is very good at delivering voice services. Early EON As for what else the system has accomplished, there are several untold stories. One is its leadership role in Cox’s Extendible Optical Network (EON) initiative.

The EON project is a catch-all term for plant upgrades and enhancements to the video-on-demand and high-speed Internet platforms. While corporate-driven, this initiative is a good example of Atlanta’s collaboration with the field.

"One of the things I like about (Cox SVP Engineering and CTO) Chris Bowick and his team is that they really…try not to do things in a vacuum," Guerrero said.

In this case, the EON/upgrade discussion and decision to invest in OC/PV took shape the summer of 2005, and that led to six to eight months of planning. "It felt good to be called in early and given some consultation opportunities, and at the end of that, getting the first infusion of finds, we all felt very proud about that," Guerrero said.

One of the upshots is that while Orange County is known to be splitting a lot of nodes, there’s a bona fide network upgrade under way. Two of Guerrero’s lieutenants, Mark Johnston, manager, network operations, and Chris Defazio, director construction services, have 1 GHz in their current job description. (See sidebar.)

Whereas 1 GHz upgrades remain a topic for discussion elsewhere, industry historians may look back to this system as one of the first sites where that talk turned into action, much as we do now regarding cable telephony. Enterprise services A press release issued from Rancho Santa Margarita this March described how Cox Business Services had expanded its service area to downtown Los Angeles.

The move may have caught someone with a legacy mindset off guard. (Is that part of Orange County’s franchise? Does Time Warner know about this?) But it was nothing out of stride for Cox, which long ago ceased to be a traditional cable television operator.

"It’s really just another couple of POPs (points of presence)," said Dave Montierth, VP of CBS in Orange County.

As bold as this strike into Los Angeles may appear to an outsider, the real gutsy move actually took place nine years ago, when Montierth first built a plan to get into the business IP, voice and transport markets. "Pat Esser initially approved our plan. He was region VP, over Leo (Brennan) at the time."

"We got the nod from Pat who said ‘If you really think you’re going to be successful, go do it,’" Montierth recalled. (Ed note: Brennan is now region VP and GM in Las Vegas; Esser is president of Cox.)

Montierth said he got nervous totaling those projections, but now thinks he "probably undersold it, initially." As of March, CBS said it was delivering commercial broadband and telecom services to 12,600 businesses in south and central Orange County.

CBC’s work with the Capistrano Unified School District, the largest employer in south Orange County, is only one of several recent showcases. The district worked with Cox to upgrade its 1.5 Mbps connectivity between its headquarters and 56 campuses to 150 Mbps.

Orders-of-magnitude gains in efficiency have a way of capturing a customer’s attention. "They can get more economic and strategic," Monteirth said. "They are so excited."

The prospects for Cox remain exciting, too. "The business community needs more reliability, faster communications and choice," Montierth said. Data mining One System-of-the-Year criterion is innovation in testing and development. It doesn’t take much encouragement to get Guerrero (who’s a self-confessed ‘nut’ when it comes to cycling metrics) talking about his efforts to exploit network and service performance data to the fullest extent possible.

"I feel like we’re on the cusp of something pretty huge," he said.

The story goes back at least a decade, to the industry’s monthly outage minutes (MOM) reports. Several years later, operators began storing those data, which lead to the ability to determine the reliability of services. The next step was to capture data from customer premise equipment, all the better to determine network health.

"Once you start storing the right data and getting a lot of people looking at it, you probably can get some actionable items," he said. "You go to the doctor before you’re dead. This is the same thing."

That engineers at corporate and system levels across the industry and vendors of various shapes and sizes are aiming for the same type of proactive solutions takes nothing away from this particular effort. It’s a live problem that needs a practical answer. Solid workforce and plant Another metric that sets OC/PV apart is workforce satisfaction. In the company’s 2000, 2002, 2004, and 2006 employee opinion surveys, OC/PV achieved the top results of all 28 Cox systems for overall satisfaction, with above-average scores in each category.

Backing up those numbers is an exceptionally low workforce turnover rate of only 10 percent. The implications for service, technical training, and overall network efficiency are significant.

That’s not say Orange County, or Cox, are models of perfection. Reflecting on the launch of data and voice a decade ago, Guerrero said taking risks is still important. "I think we swung the other direction, to where we were a little bit more cautious about things like EOD (everything on demand)," he said. "We’re trying to find that happy medium."

A digital simulcast project, which in the best of circumstances has ways of revealing hidden blemishes and loose F connectors, generated an uptick in service calls. "The good news is that we’re back to pre-simulcast levels," said Duffy Leone, region VP and general manager of OC/PV.

That is very likely a low baseline. Take it from someone whose job it is to know such things. "The California properties are generally, extremely high quality," said Jonathan Kramer, founder and principal attorney of Kramer Telecom Law Firm. "I would rank Orange County a micron above San Diego."

"But that’s comparing Cox to Cox," Kramer continues. "If you’re comparing Cox to most of the major industry players, there’s a gulf."

Kramer’s opinion is that, unlike other operators, Cox correctly views the physical plant as a revenue center. Be that as it may, the track record of this Cox system suggests a strong and enviable correlation between a sound plant, bold engineering leadership, satisfied customers and an effective, stable and enterprising workforce. • Jonathan Tombes is editor of Communications Technology. Reach him at [email protected].

Fiber: 1,100 miles; Coax: 3,500 miles

Facilities: two MTCs, four hubs

Employees: 900

Mark Johnston Manager, Network Ops, HFC Network Ops, 1GHz Upgrade Sweep

Tim Chia Director, Network Planning, HFC and Fiber Engineering & Ops, Telephone Transport and Switch Engineering & Ops

Bill Hutchins Palos Verdes, MTC and Data Ops, HFC Network Ops

Larry Jehlen Manager, RF Engineering

Dennis Nguyen Data Network Engineering Manager, Data Engineering & Ops, Network Management Systems

Chris Defazio Director, Construction Services, Residential and Business HFC Construction, 1 GHz Upgrade Construction

Ralph Frid Senior Project Manager

Mona Roderfeld Director, Regional Operating Center, Telephone Back Office (Supports Las Vegas and Santa Barbara)