How the co-location scam unfolded

What the whistle-blower wrote, using a pseudonym Ken Fong, was highly technical but explosive. It described in great detail a sophisticated market manipulation operation, at NSE’s Colo facility, that had gone on for over three years between 2011 and 2014 ‘with collaboration of NSE data centre staff’.

A SCAM UNFOLDS

Little did we know that the brown paper envelope that landed on Sucheta’s desk on a cool afternoon in January 2015 would, over the next two years, rip open the cosy cabal of market manipulators that functioned right inside India’s biggest exchange. The envelope contained a letter to BK Gupta, SEBI’s deputy general manager, in the market supervision division. It was copied to Sucheta.

What the whistle-blower wrote, using a pseudonym Ken Fong, was highly technical but explosive. It described in great detail a sophisticated market manipulation operation, at NSE’s Colo facility, that had gone on for over three years between 2011 and 2014 ‘with collaboration of NSE data centre staff’. “The NSE’s management team have chosen to hush up the matter,” he alleged. Fong claimed that he worked in the technology team of a Singapore-based hedge fund, with a large exposure to Indian stock markets. Its operations employed complex algorithms or algos⁽¹⁾, which are the heart of superfast automated trading.

NSE runs two data streams to disseminate market information. One is called snapshot, which goes out to retail/wholesale brokers and TV channels and, eventually, to investors every few seconds. This contains pending buy and sell orders and the number of shares against those orders. The second stream of information, called tick-by-tick (TBT), is far more granular and contains every single order and trade, and is disseminated to large investors at the Colo servers located at NSE’s premises. These servers are available on hire for a steep rent⁽²⁾. So, by default, information is received by high-frequency traders in the Colo server farm, a few seconds before other investors.

Fong alleged that, within the Colo facility, some traders were getting market data ahead of others by a few microseconds⁽³⁾ due to the limitations of the TBT system. This was in connivance with NSE’s technology staff. TBT, which spewed new orders one by one, was based on old technology called TCP/IP and it offered data on a first-come-first-served basis. Some traders, with the help of NSE staff, ensured that they were the first to log in every day. The firm connecting first to the server would get price information ahead of the others and thus enjoy an unfair advantage. This cycle would be repeated for every bit of price information, putting that firm ahead of others throughout the day. No global exchange in the world was using this architecture; they used an alternate architecture called multicast, which ensures dissemination of data to all users at the exact same time.

How did this happen? Well, NSE started Colo in a big hurry in 2010. As we have explained in Chapter 11, it had no regulatory approval to do so. In its hurry to grab revenues, NSE put together a patchwork solution with many holes. It was a system readymade for misuse, and some brokers quickly realised this with the help of NSE’s favoured advisers. For one, the system had to cater to a significant load, which was not budgeted for. When the demand for Colo shot up, the Exchange added multiple new servers to increase capacity to allow more traders to connect. According to Fong, the load was not the same across servers. There was a higher load (longer queue to connect) on some servers and a lower load (shorter queue) on others. If a player knew which server had lower load, he could connect to it and be ahead of the rest in getting price information. This allowed the broker/trader to mint money, since the volume of orders executed per millisecond was gigantic.

Here’s how it worked. If server 1 had 20 users and server 2 had 50, all those connected to server 1 would get faster price quotes and be able to execute trades much faster than those connected to server

2. The technology did not allow one to hop across servers during the day. “It thus became crucial to be the first one to connect to a lowest load server,” at the start of trading, explained the whistle-blower. NSE’s system was built to handle two more issues. It had a backup server, just in case one of the servers failed. If the primary server failed, everyone connected to it would be automatically switched to the backup server. NSE was also required to ensure that access was evenly distributed across servers. So, the server that each trading member would connect to was decided in advance. The Exchange knew “the number of connections on a server and (would) try to keep the load constant across servers apart from planning capacity to handle the load and, of course, invoice its members. If anyone tried to connect to a different server, NSE would be able to track it and would call up the member and ask him to fall in line,” says Ken Fong’s letter.

Within a year of NSE starting its Colo services, the smart guys had “figured out that the way to game the system lay in being the first one to connect to the server and preferably a server which was the fastest. A server could be faster due to a lower load (20 v/s 50 guys logging in) or it could be that the hardware of the server was slightly more powerful,” wrote Fong. He alleged that Sanjay Gupta of OPG Securities was one of the key players who exploited the system. Our investigation showed that OPG’s profits skyrocketed in 2013-14 (see chart). An income-tax raid on his house, in November 2017, revealed a stash of Rs11 crore (Rs110 million) in cash. His brother was known to be extremely close to the son of a powerful Union minister in the UPA government.

Gupta was not a technical whiz but apparently very good at building ‘informal relationships’. During his interactions with the NSE data centre team, he realised that he needed someone who knew the nuts and bolts of the system. He was working with Omnesys for his algos; so he managed to get a crucial person working in NSE’s Colo installations ‘on his payroll’. The TBT connection was, typically, established an hour before the market opened, but the precise time wasn’t fixed. Gupta quickly ‘warmed up’ to people at NSE’s data centre, who let him know the time at which the servers would be started so that he was the first to connect ⁽⁴⁾.

Soon, others realised that it was crucial to know when the servers would fire up and wrote a software code that would continuously check if the TBT server was started, so as to connect ahead of others.

That is when Omnesys launched a managed data centre service for its clients, claiming that it provided performance far superior to any other algo service available. Omnesys was so confident about putting clients ahead in the queue that it charged a 20% profit share from its clients. Intriguingly, the algo system of Omnesys generated a profit on every trade! Anyone who has even a rudimentary knowledge of trading would know that this was simply impossible.

Even Omnesys knew this; so it billed the profit in terms of man-hours for managed services every month and kept it variable. “Everyone just believed it was a difference in their software which gave them an edge whereas it was simply front-running,” wrote the whistle-blower. At this time, NSE’s reputation was at its peak as a professional, competent and highly profitable entity. The sordid underbelly of the HFT business was kept well-hidden through glowing media reports and its immense influence with the finance ministry, senior bureaucrats and academic institutions. When Omnesys started selling its disguised front-running services on a large scale, Sanjay Gupta was worried about losing his ‘competitive advantage’; so, he built an in-house software which no one else would have access to and hid what he was doing, says Fong.

To match a bid/offer price using algos and push out an order takes 50 microseconds in an efficient system and 200 microseconds in an inefficient system, according to the whistle-blower⁽⁵⁾. By contrast, the time taken at NSE to receive an order and send out the confirmation took ages—from 2 to 10ms (milliseconds)⁽⁶⁾. Clearly, the order processing speed was not as important as the ability to game the system. “If you could connect first to the lowest load server every day, you would be 100 times faster than the average person on co-location.” What about all the other investors? Fong says, “…Well they were the people who never even reached the stadium where the race would take place.” Most ordinary investors are clueless; they are not even in the price race of the modern trading system. Converted into profits, the advantage of being ahead by 20-50 milliseconds (one fortieth of a second) ran into hundreds of crores of rupees. About OPG Securities, the whistle-blower says, “I could tell you that it was worth about Rs2 crores per month at least or roughly a Rs100 crores⁽⁷⁾ in three years though the balance sheet of Indian firms never seem to indicate such figures probably due to tax juggleries. The quantum in case of Omnesys is easier to guess—you simply need to know the figure of managed services invoicing and multiply by 5.” But this game of making super profits through faster access was now unravelling. Others figured out OPG’s success formula and he now needed a new trick. “This is where NSE co-location staff proved very helpful. They would grant him access to the time they would switch on the servers and would give him access to servers which were the latest and least crowded,” wrote Fong. There was another significant clue to NSE’s mischief. The Exchange needed to invest a few thousand dollars to synchronise time across the Colo server farm with the Exchange clock, right down to the millisecond. But it did not do it. Hence, every computer had its own timestamp ⁽⁸⁾ and nobody could prove ‘latency numbers’. “This gave OPG a solid lead all through 2011 and most of 2012,” says Fong.

“NIRVANA IN TECHNOLOGY”

In October 2012, Ravi Apte retired as the chief technology officer. Apte, under whose charge the flawed HFT system came up, had claimed in an interview that NSE had ‘attained nirvana’ in technology. “Trading speed on NSE is close to the speed of light. And this is the limit,” he said⁽⁹⁾. Was Apte unaware of how rickety NSE’s system was and that it was being misused by a select few? But by the time Apte left, complaints about the unfairness of NSE’s system were piling up. Umesh Jain joined as chief technology officer at NSE in October 2012 and after meeting a lot of players, discovered the problems and initiated re-engineering of TBT architecture based on multicast technology. He also put in, what is called, a load balancer⁽¹⁰⁾, so that the load was evenly distributed.

Umesh also implemented NTP (network time protocol), which ensured that all servers at NSE have the same time, to microsecond accuracy, and also that the same facility was available to all members in Colo. This ensured that audit trails from different servers can be analysed collectively, in case of any issues in sequencing. Since earlier orders got matched ahead of the market, it was obvious that something was improper. NSE could not explain it away as random differences or the abilities of different software. Armed with the logs (though not necessarily understanding why this was occurring) different firms started approaching NSE to question the sequence of events.

Umesh’s moves upset the gravy train of Omnesys and those who had gamed the system. Fong alleges that OPG switched gaming of access to backup servers, which were equally powerful but were supposed to be used only if the primary servers had a problem. But this, too, was brazenly misused. The backup servers with zero load provided latency advantage and the gaming continued. Around this time, a couple of global players went live at NSE’s Colo server farm. Soon the re-engineering of TBT was complete and multicast TBT rolled out. This was much faster than before but, more important, it provided data to all traders at the same time and was the death knell for Omnesys. According to the whistle-blower, suddenly the market share of OPG fell off the charts and Omnesys’s managed services lost their advantage and went off track. Curiously, NSE had already dumped its stake in Omnesys on to Thomson Reuters! In the four months after she got this letter, Sucheta showed it to select technical experts connected to the market to understand issues; she tweeted about it without revealing details, and also shared it with sources in some Central government investigation agencies. A couple of overseas traders responded to the tweet and met us to describe and corroborate how NSE dealt with algos and the Omnesys advantage! But as Michael Lewis’s book Flash Boys says on the jacket: “Now, the world’s money is traded by computer code, inside black boxes in heavily guarded buildings. Even the experts entrusted with your money don’t know what is happening and those who do aren’t about to tell—because they are making a killing.” This was NSE’s trump card.

When Sucheta sent multiple messages to Ravi and Chitra asking for their view on the whistle-blower’s letter, they simply ignored them. Remember, NSE had already received the letter from SEBI and provided a written explanation. So arrogant was its top management that it did not even see the need to deny the contents, as it had done in their response to SEBI. We then published the entire letter⁽¹¹⁾ on www.moneylife.in on 19 June 2015 with the intention of crowd-sourcing additional information. In early July, Sucheta wrote a column for Moneylife magazine, saying that the finance ministry has suggested that, apart from SEBI, RBI should also take a detailed look at the implications of continuing HFT without adequate safeguards. Remember, NSE had a great equation with P Chidambaram, but he was no longer the finance minister.

“NSE’s management of HFT servers in the initial years until 2013 (which are the subject of the whistle-blower’s letter) may need a detailed review by SEBI or an investigation agency,” said Sucheta’s government source, off-the-record. In fact, RBI’s Financial Stability Report, released in June 2015, also identified algo trading as an area of concern with a detailed discussion. Clearly hidden from the public were NSE’s deep links with bureaucrats in New Delhi. NSE was confident that its attempt to steamroller us with a defamation case would be fully backed up by North Block, while SEBI maintained its usual silence. Events, of course, took a dramatic turn.

Extracted from Absolute Power: Inside story of the National Stock Exchange’s amazing success, leading to hubris, regulatory capture and algo scam. Written by Sucheta Dalal & Debashis Basu. 294 Pages, KenSource, 2021.

FOOTNOTES

1. Algo trading or algorithmic trading is automated trades where computer programs are written to detect tiny profit opportunities and execute extremely large trades within milliseconds or nano seconds based on pre-defined parameters, often using machine learning or artificial intelligence. The efficiency of these trades depends on the proximity of brokers’ servers to the exchange.
2. The challenge to NSE’s tech team was to generate this stream as a broadcast, which it managed sometime in 2014. It was called Multicast TBT.
3. Front-running, in market parlance.
4. From Ken Fong’s first letter to SEBI.
5. This means one-fifth of a millisecond since 1,000 micro seconds is one millisecond and 1,000 milliseconds make one second.
6. This was proportional to the load on the Exchange’s systems. The average time has been reducing from 10 milliseconds in 2010 to 2 milliseconds in 2014 as NSE improved its technology. The order of variation between different TBT servers ranged from 5ms to 50ms, which was a function of the load on the TBT server. Further variability would be based on the sequence in which traders connected to a TBT server.
7. Rs100 crore = Rs1 billion; Rs2 crore = Rs20 million
8. Latency is the time taken for order matching and confirmation of trade after a client keys it in.
9. Business Standard: 21 August 2012
10. Load balancers manage load across systems disseminating data in order to ensure consistent response time to all market participants in order to achieve fair and equal access. In effect, load balancers distribute trading volumes evenly, automatically and instantly across all servers so that no single trader got an advantage.
11. Moneylife: “Blowing the Whistle on Manipulation in NSE”: https://www.moneylife.in/article/blowing-the-whistle-on-manipulation-in-nse/42337.html