Social Media, Real Time AI and the search for Alpha

Earlier in the week at JAX Finance I gave a talk titled, "Social Media, Real Time AI and the search for Alpha". The talk looked at the general problem of extracting information from social media and some of the challenges arising from delivering a real-time system to trade off.

As a backstory to the talk I use the Yedup story to place the talk into context. Yedup are a leading company in this space who have shown it is possible to detect and disseminate real, actionable alpha.

The talk was rescheduled late in the day so some people missed it. Here are the slides:

Here is the talk abstract for those who prefer not to slide flick right now:

“The continual search for alpha in trading today has lead to an explosion of interest in alternative information sources, such as social media (Twitter, Stocktwits and so on), and the use of artificial intelligence to make sense of it all. In the world of conversations and opinions it’s hard to sift through the noise and filter out the real events that are impacting the markets. Data mining, natural language processing and heavy retrospective analysis has shown there is valuable information, but there are significant challenges finding that information in real time. Working with Yedup we’ve built an AI system that can extract real-time alpha and provide an API feed of hundreds of stocks and their related companies and industries. Used by one of the worlds biggest market makers we talk through some of the architectural choices and approaches needed to realise the real-time AI found in the Yedup system.”

The full programme description was:

Building any AI system is hard, but building a real-time AI brings its own challenges. At Yedup and Clearpool, we built a real-time AI system to simultaneously trade against hundreds of securities by extracting fundamental signals from real-time social media feeds. We’ve gone back to first principals to build out a technology stack from scratch to meet our specific needs.

This talk looks at how we’ve exploited algorithmic architecture to build a real-time AI system that delivers market leading alpha. That’s interesting in itself, but what’s more interesting is looking at some of the challenges that we’ve had to overcome in order to deliver a system that can not only trade hundreds of instruments simultaneously, but that can also correlate the relationships across industries and sectors to extract yet more alpha.

These are issues that would be applicable to any similar class of problems (such as real-time AI) and this talk explores a couple of the key challenges, such as maintaining and recovering with huge amounts of in-flight state to deliver fast, scalable and robust AI systems.