In this blog post, continued from Part 1, we’ll continue our recap of some of the topics discussed in the recent FISD webinar on Developing Software with Market Data, in which Sinara participated.
In Part 1, we looked at the critical elements you need to consider when building software with market data and the skillset your development team or supplier needs to have. We’ll now address the issues to think about when replacing a legacy system and touch on the shift to cloud and how that affects your choices.
Dealing with legacy systems
If you are looking to replace an aging or legacy market data system (or indeed other business IT system), one of the first questions you will be asking yourself is whether you even need to develop any software at all. When considering the decision on whether to ‘build v buy/off-the-shelf’, there are often good reasons to build: the software may not exist in quite the way you want, the costs or licence terms for customising an existing off-the-shelf system may be prohibitive, there is too much clutter with features you don’t need, and of course, why would you want to use exactly the same system as your competitor if you can get an edge by building something better?
Regardless of whether you develop the software or use an existing solution, a key consideration is how it will integrate with the rest of your systems. Are the data formats known about? Will you be able to convert data produced by the new system into a form that can be fed into the others? If there is already an existing database with historical data, can that easily be migrated? Does it need to be?
The other issue to consider is how you will migrate to the new system. Can you run both side-by-side for a period and perform validation testing? Can you do a ‘big bang’ replacement, or will a phased migration be needed? What environments are needed for testing and pre-production?
One key motivator for replacing many legacy platforms is to gain better access to the data they contain, which can often be quite valuable in the right form. The last thing you want is to introduce a system that once again locks away your data.
Last, but certainly not least, consider where the new system will be deployed (on-premises, cloud) and how it will be supported. The latter is a significant part of our work at Sinara, with our professional support service delivered with every system we have built.
Building applications in the cloud
No discussion on software and market data can now be had without covering the ongoing march of cloud. One of the main drivers of cloud adoption in general has been from the perspective of infrastructure and cutting costs. While this goal is still there, many financial institutions are now thinking about how the cloud can help drive innovation and create entirely new solutions for their customers that can give them a competitive edge. For example, the cloud can increase the potential of data analytics by overcoming the storage limitations of on-premise servers and enabling new AI/machine learning algorithms to be adopted, e.g. for risk analysis. The cloud also makes it easier to use sandbox tools to reduce time-to-market for new products and services or test system updates before they are deployed.
Market data is a particularly good place to start with any transition to cloud for a financial institution, since customer-specific content is not being transferred and rules around data privacy that could hinder adoption do not apply. Also, data volumes are large and require significant hardware capacity, something which is ideally suited to the cloud.
Historically, firms wanting access to streaming market data for their applications would have to make large investments in on-premises or data centre servers and network infrastructure, with enough capacity to handle large volumes. The often mission-critical nature of the applications consuming this data meant that only on-premises, dedicated hardware and IT administrators could offer the degree of resilience needed. Sinara have certainly built software for quite a few of these large-scale systems over the years.
While some of these ultra-low latency applications might not be migrating to shared cloud infrastructure anytime soon, there are many areas where latency requirements are not so critical. Given their nature, reference data, end-of-day pricing and historical tick data are relatively straightforward to migrate. By moving this kind of market data production and consumption to the cloud, financial institutions can not only reduce costs, but derive benefits from data-driven insights and offer new types of solutions to their customers. Exchanges can deliver new kinds of feeds, portals and APIs that consume these large datasets. Brokers can offer new trading and post-trade solutions, such as analysing liquidity statistics and other real-time insights. While all this could be achieved through a ‘traditional’ deployment, the cloud can considerably speed up delivery.
However, just because you’re no longer maintaining the servers in-house doesn’t quite mean you can stop thinking about them. You still need the right architectural design for the software so that it can harness the potential of the cloud; you need an application support team that understands how the software is deployed within the cloud infrastructure and can make sure it’s running effectively and perform upgrades safely.
Summary
To recap, when building software that deals with market data, you need to address latency, volume, entitlements, and other related factors so that your software will reliably deliver information to your downstream systems.
To do this, you need a development team that understands these issues in depth, can get to grips with the data model, and design an information retrieval system that will be future-proof.
It’s likely the new software will be replacing something that is already there, so you need to make choices about how that can be done in a way that reduces risk and maximises the opportunity for the new system to deliver benefits sooner.
Included in these choices will be whether to embrace the cloud for the new system, and that will be driven in part by the nature of the system itself and your organisation’s strategic choices for cloud technology.
At Sinara, we deal with all these challenges and work closely with our clients on all the stages of a software project, from requirements gathering through to deployment and support. We have seen huge changes in the world of market data over the last 30 years and we continue to bring this experience to help our clients deliver the next generation of market data systems.
