Markov processes or chains may sound vaguely familiar to many from their student days, from having read that Jim Simons started trading with them, or simply that they have something that can help us with trading, making trading more probabilistic.
These chains are widely used in many fields: biology, political science, purchasing patterns, delinquent account control, gambling, etc. But… Can they help us in trading?
In this post, I’m going to provide an introduction to Markov chains to make the concept clear, first mathematically, then with some code, and finally I’ll explain how we can use them in our trading. I’ll try to keep the mathematics simple to make it manageable and so that those who aren’t accustomed to mathematics can follow along.
What are Markov chains?
Let’s start with the basics, making clear what these chains are. Without getting into mathematical terms, we can say that a Markov chain is a series of events that can occur, where the probability of one event or another happening depends on the state we are currently in.
A simple Markov chain could be the following: Let’s imagine we have three possible states for stocks: stocks that go up, stocks that go down, and stocks that remain flat. Now we would conduct a study to determine what percentage of state changes occur in each of them, that is, if the last candle is up, what is the percentage that the next one will be up, down, or stay the same?
Let’s imagine we already have all the numbers and we display them in a transition probability matrix:
| Up | Down | The same | |
|---|---|---|---|
| Up | 0.42 | 0.46 | 0.12 |
| Down | 0.45 | 0.44 | 0.11 |
| The same | 0.47 | 0.48 | 0.05 |
This mathematically is a matrix that we have seen many times, and I think it’s quite easy to understand. If the last candle is up (Bullish), the probability that it goes up again is 42%, that it goes down is 46%, and that it stays the same is 12%.
This probability matrix must comply with three very simple rules:
- It must be square (in this case it’s 3×3)
- The values must be between zero and one
- The sum of each row must equal one
This matrix can also be represented in another way with a digraph.
With this, we have at a single glance a much more visual way of seeing the connections between one state and another. We also clearly see if we are in the “Up” state what the probabilities are of going to the next state.
Calculating the probabilities
At the beginning we will have three states. The ones we saw in the previous image, but now we are going to represent it mathematically in a vector where the first position will be “Up”, the second will be “Down” and the third will be “Same”:
This could be understood as for the “Up” position, being [1 0 0], we will have a 100% chance that it goes up and 0% that it goes down or stays the same, and the same with the rest of positions. In this case, since it’s a known state, the probabilities are 100%, because logically we already know what the initial state is.
We will also convert the table we saw earlier into a probability matrix. That table seen from a mathematical point of view is:
To apply Markov what we need to do is a matrix multiplication between the initial vector and the probability matrix. In this example we are going to assume that the position we are in is that the last candle is down, so we are going to start with the “Down” state (or also [0 1 0]). So we will do the multiplication like this:
As a quick reminder of matrix multiplication, we’re going to say that what we need to do is multiply [0 1 0] by each column (first number by the top one, second number by the middle one, and third number by the bottom one) and add the values it gives us in each column. That is, in our case it would be something like this:
Then the normal sum of each column is performed and thus we will have the probabilities of each state.
This we could say is the first prediction we make to know the probabilities of the next state. In this case, the probabilities that the next position will be up are 45%, down 44%, and that it stays the same are 11%.
But what if we wanted to know the probabilities of another state further ahead? What are the possibilities within two days for each candle?
Well, to do this it would be the same but the starting vector would be the one we just calculated, since we start from the state we have tried to predict. So the equation would look like this:
The steps to follow are the same as in the previous operation, so we need to multiply the new vector by the probability matrix we already knew:
And with this we will have the probabilities of what the candle might be like in two days:
If we wanted to know what the probabilities would be in a third state, fourth or fifth, it would be the same, always matrix multiplication with the previous state.
And this is everything necessary to understand Markov chains from a mathematical point of view. Now we could even calculate with pencil and paper the probabilities of the next candles, but for us it’s not necessary since we have software that can do it for us in a faster and simpler way. Let’s see it.
Python implementation of Markov chains
We can see Markov chains with the pydtmc library, and within that library the MarkovChain class. Let’s see how this library works:
from pydtmc import MarkovChain
We directly instantiate the class that interests us since pydtmc brings many other functionalities that we are not going to use.
Then we just have to create our probability matrix and pass it directly in the class initialization, along with the names of each state.
matrix = [[0.42,0.46,0.12],[0.45,0.44,0.11],[0.47,0.48,0.05]] markov_chain = MarkovChain(matrix, ['Up', 'Down', 'Same'])
The simple way to see what the probabilities of the third step are is:
print(cadena_markov.redistribute(3,'Down')) # Output: [0.438997 0.45309 0.107913]
Here we pass the number of steps we want for the predictions (in this case the third day) and the initial state where we want to start (in this case ‘Down‘), with this it will give us the distribution of the next following events.
This looks easier now, and you don’t need to know much mathematics; having the probability calculations is enough. However, calculating these probabilities is something you have to work on, although it’s not difficult to know the probabilities of one candle after another with a simple Python script.
Possibilities in trading
There are many ways to take advantage of this simple form of Markov chain (there are actually more ways and the topic could go on for a while), here we are going to present some small brushstrokes so you know how probability studies can help you in trading.
It’s clear that if we only look at individual candles the possibilities are practically 50% that it goes up or down, that is, like flipping a coin. But what would happen if we started looking at two or more candles? What if we start looking at patterns? In these cases the probabilities would be very different and the cases would be a bit more complex. The digraph could be different and more complex. There wouldn’t be a total connection between all possible states and surely the probabilities of going from one state to another would change.
What if with technical indicators we extract our own states (imagine the states and then look at the probabilities of change between several of them over the days)?
As you can see, just by thinking a little, many other ways of being able to use the simplest form of Markov already appear. The rest I leave to your imagination and for you to entertain yourselves extracting probabilities from your datasets.
Conclusions
Although this example might seem a bit silly, I believe it was necessary to properly understand what a Markov chain is and why it works the way it does. Markov chains can still be much more complex and other characteristics can even be added to them, such as rewards, bringing us much closer to reinforcement learning, but I’ll explain that another time.
Markov chains are also key to understanding other processes that we’ll also see later, such as the Monte Carlo method, but don’t think that the topic of Markov chains ends here. I invite you to continue investigating a bit more about this topic and other equations that derive from it. It’s a very interesting topic that can give you many ideas to apply in your trading.
As always, any doubts or clarifications, don’t hesitate to put them in the comments or send me a message and I’ll respond as soon as possible.
