Category Archives: Data Integrity Management

Getting Closer to a Reliable FAR

OK finance teams, time to come out of the shadows.  At most operators with whom I have worked, the focus of enterprise data quality efforts has been on optimizing network operations.  Misalignment between the network and data in systems that support planning, provisioning, activation and service assurance adds friction and cost to essential telco processes.  No new insight here.

Lately I’ve been spending more time with the finance guys.   Despite stereotypes to the contrary, they’re not just number crunchers.  They care about what’s in the network- where it is, how old it is, the condition it’s in, where it’s been, where it’s going and what happens at end of life.  Complicating their lives, the financial database of record for network assets, the Fixed Asset Register (FAR), typically suffers from the same data issues confounding the Ops teams—if not worse.

Sounds bad, but Finance doesn’t have to worry about delivering or supporting new services.  So what’s the harm?   Based on the earful I’ve received from finance organizations, including a Tier 1 CFO—plenty.  This diagram provides a sampling of corporate functions dependent upon accurate asset records in the FAR:

Mission Critical Role of the FAR

Role of the FAR

Among the potential costs of inaccurate fixed asset records are:

  • Improper calculation of depreciation
  • Failure to identify impaired assets and candidates for accelerated write-downs
  • Overpayment of property taxes
  • Overpayment of insurance premiums
  • Restatement of past financial results
  • Risk of regulatory penalties
  • Exposure to fraud, theft and asset mismanagement cases

Such issues get corporate board-level attention.  I am aware of several recent cases of poor FAR audit results prompting an operator to launch a FAR cleanup effort or even a full asset management program.

To their credit, financial reporting analysts I have spoken with are not blind to their data woes—just typically dependent upon compromises and work-arounds.  They often manage as best they can by mining numerous B/OSS’s to cobble together a view of assets across all network layers and asset classes.  Gaps and inaccuracies in this view abound.

Among the most common methods finance organizations employ to address the situation are manual audits performed on sample sites once or twice a year.   This mostly provides insights on how far off the FAR is from reality.  Generally, such spot audits are too limited and expensive to support systemic and continuous correction of the errant data.

So how do we achieve a reliable FAR (before the Board takes it up as a crusade)?

It starts with determining the impact of inaccurate asset records on financial reporting and planning, corporate governance, taxation and regulatory compliance.  Is the exposure minimal and bounded, or are the risks unacceptable?  Assuming the latter, a FAR get-well plan should include:

  • Data acquisition methods for both active and passive network assets that use the network itself as a source versus other systems
  • Comparison of this data to the FAR and reconciliation of errors
  • A permanent mechanism to keep the FAR aligned with the changing network so the data stays clean
  • Commensurate process enhancements and guardrails to reinforce automated approaches—which can never guarantee 100% accuracy on their own

When tightly aligned with the network, the FAR becomes more than a reporting tool, it can become a strategic enabler for Capex optimization.

Making the Connection

It’s 7AM and I can’t put off getting up any longer, so I look out the window and see there’s a light frost on the grass, which the weather channel warned me about 3 days ago.   An hour later I’m at the train station waiting for the 7:42 which is delayed because the frost has caused the points to seize up in a town 50 miles away, so now the entire South East rail system is completely snarled up. They predicted the weather and probably knew there would be a point’s failure, but still the network crashed. So I need to phone work to let them know I’m going to be late, but I can’t connect. Thousands of other commuters around me are also trying to phone ahead, but the network can’t handle it. It seems to happen every day.

We are surrounded by events which are beyond our control, but often they happen in predictable ways. The points failure was perhaps less predictable than my alarm, but we always knew that when the temperature dropped there would be some kind of failure somewhere that would lead to cancellations and a breakdown of the network. We always knew that rush hour would become an agonising crawl into town on overcrowded trains. The congestion could probably have been avoided if they could have predicted which parts of the network were under the most stress and the impact on the network in the event of failure or congestion at those stress points. Additional resources could then be provided at those points, or alternative routes planned to bypass the congestion and limit the ripple out effect, like a fire break.  The problem is only likely to get worse, and the network more unreliable, as the population increases and more people than ever rely on the rail network to get to work.

With the arrival of LTE and rapidly increasing popularity of Video on Demand then telecoms networks are also facing increasing levels of congestion and instability. Global data traffic is predicted to increase by 10 to 20 times by 2019 (Cisco).   In order to meet regulatory obligations and maintain customer experience Capex is set to spiral upwards. MNOs, who are already facing a year on year decrease in ARPU, will struggle to keep pace with demand and the risk of congestion will be ever present.

As with rail networks MNOs need a longer term strategy in place to understand where and when future choke points in the network will occur so that the risk of congestion can be eliminated for the least cost. Subex Capacity Management provides the capability to predict these points of congestion by monitoring and correlating metrics from across the network to provide detailed forecasts of network utilisation. Additional factors can be brought into the forecasts, such as the impact of major events or the rolling out of M2M services and different scenarios played out to understand how the network will respond. By automating the forecasting process network managers can be alerted long before issues become critical and congestion begins to occur. They can evaluate different options for either re-homing traffic or augmenting the network for the least possible cost. Stranded or un-utilised assets can even be recovered and re-located to satisfy demand for very little cost.

CFOs need to find ways to keep increasing revenue while controlling costs, and CTOs need to keep network delivering ever greater speeds as volumes of traffic increase exponentially.  Both need to look into the future to avoid a future of network instability, falling quality, crippling network costs and lost revenue.

Network Discovery and Analytics – The Evolution (Part 2)

In continuation to my previous blog on Network Discovery and Analytics, which mostly revolved around  some of the core but basic functionality that a Discovery system needs to possess. In this post lets discuss some of the key challenges that discovery systems have to face when they first get deployed.

1)      Resistance from network operations teams.

  • Network operations for example would like for the network not to be disturbed during peak traffic hours or during maintenance windows. Towards that intent, the Network operations teams would like to be confident that a discovery system can be set up such that the system will at no cost touch the network for a configured duration. The network operations team would expect to have confidence in the discovery system that irrespective of the stage at which a network polling activity is in, the activity needs to be suspended on the event of hitting a black out window.
  • Non – intrusive discovery is what a network operations team would be confident about. By non-intrusive discovery, I mean that a device which is already overloaded with respect to its resource consumption should not be further affected by querying for information to perform a discovery

2)      Resistance from IT teams managing N/EMS systems.

  • In the event of discovery from NBI of N/EMS or from gateways that maintain stateful sessions, in a lot of situations it would be required of the discovery system to not consume more than a configured number of sessions. This may be required to make sure that other northbound OSS systems a not denied a session request from the N/EMS or gateways. It may also be because only a few NBI sessions have been purchased from the equipment vendor and using more number of sessions than purchased may lead to a contract violation or error in discovery.

The OSS Network discovery applications have to evolve and stand up against the above challenges by introducing functionality like blackout support and network hit throttling capability.

Moving forward to the next stage of the evolution. Tier-1 or Tier-2 operators have discovery systems deployed and use this information to keep inventories in sych with the real world. But inventories typically store services or end-to-end circuits. And what is discovered are individual components of a service or circuit which have been provisioned within a network element. It is important from the perspective of the service provider to view the current state of a circuit in the Network, compare it with the inventory to fix misalignment. This brings in the responsibility within the Discovery system to be able to assimilate service components discovered within each network element along with the network element interconnectivity information and be able to plot end to end circuits for various technologies.

As of today,  a few tier 1 operators who are pretty mature in their process are looking towards evolving the discovery system’s capabilities to be in near real time synch with the state of the network. A system which is able to listen to traps/events from the network and refresh its database with the latest state of the network. And use the near real time discovery system to evolve the inventories to be near real time up-to-date with the reality in the network.

It’s another thing that a lot of inventories even today are far from accurate despite a whole lot of tools out there in the market specifically designed to solve data integrity issues of Inventories. And the reason for that is the lack of a sound practice around the usage of these tools or a lack of commitment to adhere to a data integrity program or a failed OSS inventory transformation project etc

In the interim, while operators are trying to get their inventories cleaned up we believe that the Discovery system along with intelligent and actionable analytics has a lot more to offer to the planning and service assurance teams which has been mentioned below.

1)      The discovered physical fabric can be compared and/or enriched with OSS inventories and ERP systems to build a 3600 view the asset. The 3600 view when recoded for a period of time becomes a very power source to a bunch of analytical function which would provide actionable intelligence to planning and network operation teams in helping with Capex avoidance.

2)      The discovered logical/service component information when captured for a period of time can again be a source to bunch of other analytical functions (Time series trending and forecasting, What-If modeling and optimization) to help network operations and planning teams to perform network capacity management on accurate and as discovered information.

3)      The discovered information when assimilated with topological analytics to calculate end-to-end circuits/services can be a powerful source of information to Service assurance teams to overlay alarms and generate a service impact view in their NOC.

Network Discovery and Analytics – The Evolution (Part 1)

Let us look at why OSS Network discovery systems came into existence, how they evolved over years and in our opinion the areas in which it would play a critical role going forward.

Most Tier-1 and Tier-2 service providers have at some point in time invested in building one or more inventory system which would work as a central repository for functions like Planning, Design, Fulfillment and Assurance. For these functions to have a reliable and up-to-date view of the network, it is important that depth and breadth of information is captured by the inventory. By breadth, I mean the ability to store information from multiple device types from multiple device vendors. And by depth I mean the ability to store physical inventory and also logical inventory for multiple technologies like PDH, SDH/Sonet, DWDM, Ethernet, ATM, FR, IP, MPLS, xDSL, FTTx, 2G, 3G, LTE etc. These Inventories have largely been maintained/updated by manual data entry and in the event of weak process or lack of discipline in following processes; these systems would very quickly go out of synch with the network. And hence OSS Network discovery systems first came into existence in order to help keep inventories up-to-date with the real world. Towards this intent, OSS Discovery systems needed to take care of the following requirements.

1) Data that needs to be acquired from the network

  • Ability to perform shallow discovery i.e. discovery of physical fabric (Physical inventory like Racks, Shelves, Cards, Ports) from most leading-class networking equipment vendors based on standards like SNMP Entity Mib-2, TMF 814 etc.
  • Ability to discover deep logical information like vlans, ip interfaces, ip subnets etc again using standard mibs (like RFC 2233, RFC 2674), MTNM etc.
  • Ability to discover inter-connectivity (PTOPO) between devices wherever possible

2) How data needs to be acquired from the network

  • A lot of device vendors equipment cannot be discovered using standard SNMP interfaces and hence the need to be able to discover using other network management protocols like TL1, MML, CMIP, Corba, WebServices.
  • Many a times Network operations teams are not comfortable with multiple IT systems (N/EMS, Discovery, Fulfillment, Assurance etc) connecting directly to the devices for multiple reasons like worried about resource consumption on devices which may affect revenue generating traffic and access control list and login configurations on all devices need to be modified for discovery servers to access the device. This may seem like a trivial one time activity, but a few large Tier 1 operators which have strong process in place; this activity can run into a few months.

And hence discovery systems were forced to interface with north bound interfaces of N/EMS systems or gateways in order to perform discovery. These NBI APIs on a lot of occasions end up being propriety or customizations done on standard specifications.

For reasons mentioned above, discovery systems with ability in capturing breadth and depth in discovery have had to build a lot of device vendor specific adaptors instead of standards based adaptors.

Get Started with Subex