Using matplotlib to identify trading signals

Finding trading signals is one of the core problems of algorithmic trading, without any good signals your strategy will be useless. This is a very abstract process as you cannot intuitively guess what signals will make your strategy profitable or not, because of that I’m going to explain how you can have at least a visualization of the signals so that you can see if the signals make sense and introduce them in your algorithm.

We’re going to use matplotlib to graph the asset price and add buy/sell signals on the same graph, this way you can see if the signals are generated at the right moment or not: buy low, sell high.

Data preparation

For this tutorial I picked a very simple strategy which is a crossing moving average, the idea is to buy when the “short” moving average, let’s say 5-day is crossing the “long” moving average, let’s say 20-day, and to sell when they cross the other way.

First of all, we need to install matplotlib via the usual pip:

pip install matplotlib

This example requires pandas and matplotlib:

import pandas as pd
import matplotlib.pyplot as plt

I’m using the E-mini future dataset from Quandl, see this article.

Loading data and computing the moving averages is pretty trivial thanks to Pandas:

data = pd.DataFrame.from_csv(path='EMini.csv', sep=',')

# Generate moving averages
data = data.reindex(index=data.index[::-1]) # Reverse for the moving average computation
data['Mavg5'] = data['Settle'].rolling(window=5).mean()
data['Mavg20'] = data['Settle'].rolling(window=20).mean()

Now the actual signal generation part is a bit more tricky:

# Save moving averages for the day before
prev_short_mavg = data['Mavg5'].shift(1)
prev_long_mavg = data['Mavg20'].shift(1)

# Select buying and selling signals: where moving averages cross
buys = data.ix[(data['Mavg5'] <= data['Mavg20']) & (prev_short_mavg >= prev_long_mavg)]
sells = data.ix[(data['Mavg5'] >= data['Mavg20']) & (prev_short_mavg <= prev_long_mavg)]

buys and sells is now containing all dates where we have a signal.

Plotting the signals

The interesting part is the graphing of this, the syntax is simple:

plt.plot(X, Y)

We want to display the E-Mini price and the moving averages is pretty simple, we use data.index because the dates in the DataFrame are in the index:

# The label parameter is useful for the legend
plt.plot(data.index, data['Settle'], label='E-Mini future price')
plt.plot(data.index, data['Mavg5'], label='5-day moving average')
plt.plot(data.index, data['Mavg20'], label='20-day moving average')

But for the signals, we want to put each marker at the specific date, which is in the index, and at the E-Mini price level so that visually it’s not too confusing:

plt.plot(buys.index, data.ix[buys.index]['Settle'], '^', markersize=10, color='g')
plt.plot(sells.index, data.ix[sells.index]['Settle'], 'v', markersize=10, color='r')

data.ix[buys.index][‘Settle’] means we take the ‘Settle’ field in the data DataFrame

plt.ylabel('E-Mini future price')
plt.xlabel('Date')
plt.legend(loc=0)
plt.show()

Here is the final result:

Conclusion

In conclusion, you can interpret this by noticing that most buying signals are at dips in the curve and selling signals are at local maximums. So our signal generation looks promising, however without a real backtest we cannot be sure that the strategy will be profitable, at least we can validate or not a signal.
The main advantage of this method is that we can instantly see if the signals are “right” or not, for example you can play with the short and long moving average, you could try 10-day versus 30-day etc. and in the end you can pick the right parameters for this signal.

Create a trading strategy from scratch in Python

To show you the full process of creating a trading strategy, I’m going to work on a super simple strategy based on the VIX and its futures. I’m just skipping the data downloading from Quandl, I’m using the VIX index from here and the VIX futures from here, only the VX1 and VX2 continuous contracts datasets.

Data loading

First we need to load all the necessary imports, the backtest import will be used later:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from backtest import backtest
from datetime import datetime

For the sake of simplicity, I’m going to put all values in one DataFrame and in different columns. We have the VIX index, VX1 and VX2, this gives us this code:

VIX = "VIX.csv"
VIX1 = "VX1.csv"
VIX2 = "VX2.csv"

data = []
fileList = []
# Create the base DataFrame
data = pd.DataFrame()

fileList.append(VIX)
fileList.append(VIX1)
fileList.append(VIX2)

# Iterate through all files
for file in fileList:
# Only keep the Close column
tmp = pd.DataFrame(pd.DataFrame.from_csv(path=file, sep=',')['Close'])

# Rename the Close column to the correct index/future name
tmp.rename(columns={'Close': file.replace(".csv", "")}, inplace=True)

# Merge with data already loaded
# It's like a SQL join on the dates
data = data.join(tmp, how = 'right')

# Resort by the dates, in case the join messed up the order
data = data.sort_index()

And here’s the result:
[table]
Date,VIX,VX1,VX2
02/01/2008,23.17,23.83,24.42
03/01/2008,22.49,23.30,24.60
04/01/2008,23.94,24.65,25.37
07/01/2008,23.79,24.07,24.79
08/01/2008,25.43,25.53,26.10
[/table]

Signals

For this tutorial I’m going to use a very basic signal, the structure is the same and you can replace the logic with your whatever strategy you want, using very complex machine learning algos or just crossing moving averages.

The VIX is a mean-reverting asset, at least in theory, it means it will go up and down but in the end its value will move around an average. Our strategy will be to go short when it’s way higher than its mean value and to go short when it’s very low, based on absolute values to keep it simple.

high = 65
low = 12

# By default, set everything to 0
data['Signal'] = 0

# For each day where the VIX is higher than 65, we set the signal to -1 which means: go short
data.loc[data['VIX'] > high, 'Signal'] = -1

# Go long when the VIX is lower than 12
data.loc[data['VIX'] < low, 'Signal'] = 1

# We store only days where we go long/short, so that we can display them on the graph
buys = data.ix[data['Signal'] == 1]
sells = data.ix[data['Signal'] == -1]

Now we’d like to visualize the signal to check if, at least, the strategy looks profitable:

# Plot the VX1, not the VIX since we're going to trade the future and not the index directly
plt.plot(data.index, data['VX1'], label='VX1')
# Plot the buy and sell signals on the same plot
plt.plot(sells.index, data.ix[sells.index]['VX1'], 'v', markersize=10, color='r')
plt.plot(buys.index, data.ix[buys.index]['VX1'], '^', markersize=10, color='g')
plt.ylabel('Price')
plt.xlabel('Date')
plt.legend(loc=0)
# Display everything
plt.show()

The result is quite good, even though there’s no trade between 2009 and 2013, we could improve that later:

Backtesting

Let’s check if the strategy is profitable and get some metrics. We’re going to compare our strategy returns with the “Buy and Hold” strategy, which means we just buy the VX1 future and wait (and roll it at each expiry), this way we can see if our strategy is more profitable than a passive one.
I put the backtest method in a separate file to make the main code less heavy, but you can keep the method in the same file:

import numpy as np
import pandas as pd

# data = prices + dates at least
def backtest(data):
cash = 100000
position = 0
total = 0

data['Total'] = 100000
data['BuyHold'] = 100000
# To compute the Buy and Hold value, I invest all of my cash in the VX1 on the first day of the backtest
positionBeginning = int(100000/float(data.iloc[0]['VX1']))
increment = 1000

for row in data.iterrows():
price = float(row[1]['VX1'])
signal = float(row[1]['Signal'])

if(signal > 0 and cash - increment * price > 0):
# Buy
cash = cash - increment * price
position = position + increment
print(row[0].strftime('%d %b %Y')+" Position = "+str(position)+" Cash = "+str(cash)+" // Total = {:,}".format(int(position*price+cash)))

elif(signal < 0 and abs(position*price) < cash):
# Sell
cash = cash + increment * price
position = position - increment
print(row[0].strftime('%d %b %Y')+" Position = "+str(position)+" Cash = "+str(cash)+" // Total = {:,}".format(int(position*price+cash)))

data.loc[data.index == row[0], 'Total'] = float(position*price+cash)
data.loc[data.index == row[0], 'BuyHold'] = price*positionBeginning

return position*price+cash

In the main code I’m going to use the backtest method like this:

# Backtest
backtestResult = int(backtest(data))
print(("Backtest => {:,} USD").format(backtestResult))
perf = (float(backtestResult)/100000-1)*100
daysDiff = (data.tail(1).index.date-data.head(1).index.date)[0].days
perf = (perf/(daysDiff))*360
print("Annual return => "+str(perf)+"%")
print()

# Buy and Hold
perfBuyAndHold = float(data.tail(1)['VX1'])/float(data.head(1)['VX1'])-1
print(("Buy and Hold => {:,} USD").format(int((1+perfBuyAndHold)*100000)))
perfBuyAndHold = (perfBuyAndHold/(daysDiff))*360
print("Annual return => "+str(perfBuyAndHold*100)+"%")
print()

# Compute Sharpe ratio
data["Return"] = data["Total"]/data["Total"].shift(1)-1
volatility = data["Return"].std()*252
sharpe = perf/volatility
print("Volatility => "+str(volatility)+"%")
print("Sharpe => "+str(sharpe))

It’s important to display the annualized return, a strategy with a 20% return over 10 years is different than a 20% return over 2 months, we annualize everything so that we can compare strategies easily. The Sharpe Ratio is a useful metric, it allows us to see if the return is worth the risk, in this example I just assumed a 0% risk-free rate, if the ratio is > 1 it means the risk-adjusted return is interesting, if it’s > 10 it means the risk-adjusted return is very interesting, basically high return for a low volatility.
In our example we have a pretty nice Sharpe ratio of 4.6 which is quite good:

Backtest => 453,251 USD
Annual return => 38.3968478261%

Buy and Hold => 53,294 USD
Annual return => -5.07672097648%

Volatility => 8.34645515332%
Sharpe => 4.60037789945

Finally, we want to plot the strategy PnL vs the “Buy and hold” PnL:

plt.plot(data.index, data['Total'], label='Total', color='g')
plt.plot(data.index, data['BuyHold'], label='BuyHold', color='r')
plt.xlabel('Date')
plt.legend(loc=0)
plt.show()

The strategy perfomed very well until 2010 but then from 2013 the PnL starts to stagnate:

Backtest

Conclusion

I showed you a basic structure of creating a strategy, you can adapt it to your needs, for example you can implement your strategy using zipline instead of a custom bactktesting module. With zipline you’ll have way more metrics and you’ll easily be able to run your strategy on different assets, since market data is managed by zipline.
I didn’t mention any transactions fees or bid-ask spread in this post, the backtest doesn’t take into account all of this so maybe if we include them the strategy would lose money!

Load market data from Quandl

In the previous articles, we loaded market data from CSV files, the drawback is that we’d need to redownload the CSV file every day to get latest data. Why not get them directly from the source ? Quandl is a website aggregating market data from various sources: Yahoo Finance, CBOE, LIFFE among others.

Fortunately for us, Quandl has an API in Python which let you access its data. First of all, you’ll need to get your personal API key here, here is a basic code snippet:

import quandl

quandl.ApiConfig.api_key = 'YOUR_API_KEY'
VIXCode = "CHRIS/CBOE_VX1"

VX1 = quandl.get(VIXCode)

The quandl.get() method returns a Pandas data frame with the dates in the index and open/high/low/close data, this depends on the data source, you may get more information like volume etc.

In conclusion now you can directly work with that data frame, you can merge it with other data, apply some calculations and use it as an input in a machine learning algorithm. The main advantage is that you’ll always get the latest data, no need to redownload a file.

Trading with Poloniex API in Python

Poloniex is a cryptocurrency exchange, you can trade ~80 cryptocurrencies against Bitcoin and a few others against Ethereum. I chose to trade on Poloniex because it supports a lot of currencies and the liquidity is usually very good, we can easily implement an algorithmic trading strategy on this exchange.

The most traded currencies are:
– Bitcoin (BTC)
– Ethereum (ETH)
– Monero (XMR)
– Tether (USDT)

The Setup

Fortunately for us, Poloniex provides an API to get market data, to get balances for each currency and to send buy/sell orders to the market. You can find a documentation here.

I found a Python wrapper for their API on GitHub, this one is super easy to use.
You can install the package like this:

pip install https://github.com/s4w3d0ff/python-poloniex/archive/v0.3.5.zip

Once it’s installed, you need to insert the appropriate import in your code:

from poloniex import Poloniex

Now you need to get an API key in order to be able to retrieve your account balances and to send orders to the market. If you just want to get market data you can skip that part.
Go to https://poloniex.com/apiKeys , click on Create new key, now you have the API key and you may need to get some email validation to see the secret key (which you also need). Check the options you want, if you want to trade via the API, just select the appropriate check box, same for withdrawals.

Using the API

In your code, you need to set up the connection so that you can get authenticated. You can just use the commented line if you only want to access the public API:

apiKey = "API_KEY"
secret = "SECRET_KEY"
polo = Poloniex(apiKey, secret)
# polo = Poloniex()

If you want to get market data for a ticker:

market_data = polo.returnTicker()['BTC_ETH']
bid = market_data["highestBid"]
ask = market_data["lowestAsk"]
volume = market_data["baseVolume"]

Now to send an order, it’s pretty simple:

pair = "BTC_ETH"
price = 0.1
order = polo.buy("BTC_ETH", price, 1)
order = polo.sell("BTC_ETH", price, 1)

You’ll get an order object in JSON, resultingtrades is an array of trades generated by the order, the order can be filled straight away with multiple trades:

{‘orderNumber’: ‘0000000’, ‘resultingTrades’: []}

To manage your risks, you’ll need to retrieve your balances:

balance = polo.returnBalances()
print("ETH="+str(balance ["ETH"]))

With this basic API you can code any algorithmic strategy in Python for Poloniex, you can try to predict the value of a cryptocurrency using our previous tutorials for example.

Using feature selection to improve a machine learning strategy

For this tutorial, we’re going to assume we have the same basic structure as in the previous article about the Random Forest article. The idea is to do some feature engineering to generate a bunch of features, some of them may be useless and reduce the machine learning algorithm prediction score, that’s where the feature selection comes into action.

Feature engineering

This is not a tentative of a perfect feature engineering, we just want to generate a good number of features and pick the most relevant afterwards. Depending on the dataset you have, you can create more interesting feature like the day, the hour, if it’s the weekend or not etc.
Let’s assume we only have one column, ‘Mid’ which is the mid price between the bid and the ask. We can generate moving average for various windows, 5 to 50 for example, the code is quite simple using pandas:

for i in range(5, 50, 5):
data["mavgMid"+str(i)] = pd.rolling_mean(data["Mid"], i, min_periods=1)

This way we get new columns: MavgMid5, MavgMid10 and so on.
We can also do that for the moving standard deviation which can be useful for a machine learning algorithm, almost the same code as above:

for i in range(5, 50, 5):
data["stdMid"+str(i)] = pd.rolling_std(data["Mid"], i, min_periods=1)

We can continue with various rolling indicators, see the full list here. I personally like rolling_corr() because in the crypto-currencies world, correlation is very volatile and contains a lot of information, especially for inter exchange arbitrage opportunities. In this case you need to add another column with prices from another source.

Here is an example of a full function:

def featureEngineering(data):
# Moving average
for i in range(5, 50, 5):
data["mavgMid"+str(i)] = pd.rolling_mean(data["Mid"], i, min_periods=1)

# Rolling standard deviation
for i in range(5, 50, 5):
data["stdMid"+str(i)] = pd.rolling_std(data["Mid"], i, min_periods=1)

# Remove the 50 last rows since 50 is our max window
data = data.drop(data.head(50).index)

return data

Feature selection

After the feature engineering step we should have 20 features (+1 Signal feature). I ran the algorithm with the same parameters as in the previous article, but on XMR-BTC minute data over a week using the Crypto Compare API (tutorial to come soon) and I got the decent score of 0.53.

That’s a good score but maybe our 20 features are messing with the Random Forest ability to predict.

We’re going to use the SelectKBest algorithm from Sci-kit learn which is quite efficient for a simple strategy, we need to add some import in the code first:

from sklearn.feature_selection import SelectKBest, f_classif

SelectKBest() takes 2 parameters at minimum: an algorithm, here we picked f_classif since we’re using Random Forest Classifier and the number of features you want to keep:

data_X_train = SelectKBest(f_classif, k=10).fit_transform(data_X_train, data_Y_train)
data_X_test = SelectKBest(f_classif, k=10).fit_transform(data_X_test, data_Y_test)

Now data_X_train and data_X_test contains 10 features each, selected using the f_classif algorithm.

Finally the score I got with my XMR-BTC dataset is 0.60, 6% is a pretty nice improvement for a basic feature selection. I picked 10 randomly as a number of feature to keep, but you can loop through different number to determine the best number of features, but be careful of over fitting!

Common Mistakes to Avoid When Cryptocurrency Trading

This article by Steven Buchko was originally published at CoinCentral.com

Whether you’re a crypto expert or just getting your feet wet with investing, there’s plenty to be aware of when trading your way through the cryptocurrency industry. Unlike in traditional markets, cryptocurrency trading is chock full of volatility, nefarious players, and irrational price movements.

In this article, we’ll teach you about some of the common mistakes in cryptocurrency trading and how you can avoid them.

Mistake #1: Chasing Pumps aka FOMO

Probably the most common (and easiest) mistake to make in cryptocurrency trading is buying into a coin after it’s already risen a significant amount. Investors that bought into Ripple (XRP) and Tron (TRX) at the peak of their runs in 2017 definitely felt the pain just a few weeks later in 2018. It may be your instinct to throw some money in the ring when you see a coin shoot up 30-40% because it’s “hot.” Don’t.

ripple fomo chart

Extreme increases in price are almost always accompanied by some type of pullback. By the time you hear about a “hot” coin, it’s usually too late. Unless you’ve done your research, believe in the fundamentals of the coin, and want to hold it for the long-term (>1 year), wait until the pullback to invest.

Pump and Dumps

Pump and dumps (PnDs) are a special breed of pumps that are guaranteed to leave you burned. If you see an unknown coin skyrocket all of a sudden, be wary. It’s most likely part of a PnD scheme. They’re basically coordinated efforts to artificially drive up the price of a coin (the pump) before selling it to those who FOMO’d in (the dump).

When you come across a coin like this, the first thing to check is the trading volume. CoinMarketCap is a great resource for this. Any 24-hour trading volume under $1 million should raise a red flag.

Mistake #2: Not Knowing Your Investments

Don’t just blindly follow the advice of some Twitter or YouTube “guru” for investment picks. Many times, these high-profile individuals are paid to promote certain coins. Even John McAfee, one of the most well-known figures in the space admitted that he gets paid to promote projects. Question the coins that you’re told to invest in.

At the bare minimum, you should devote a half hour to researching any project in which you plan to invest in. Check out what problem it’s attempting to solve, the team building it, and the economics of the coin. Has the project partnered with anyone significant? Any notable names as advisors? These are all things you should know.

Even a quick Google search could unveil some information that turns what may seem like gold into trash. Taking it a step further, you should ideally read the white paper of each project you invest in.

bitconnect scam google search

Joining or forming an investment group can do wonders to help with this. It forces you to do research so you can explain your investment reasoning to your peers. It also puts you an environment in which you have to challenge your assumptions as others question your reasoning.

Mistake #3: Selling At Inappropriate Times

The opposite of chasing pumps, emotion-driven selling is still cut from the same cloth. It’s difficult, but you need to stay level-headed when trading – keep emotions out of it. Time and time again, coins have dipped down double-digit percentages before rocketing to 200-300% gains.

When a coin you own starts to drop in value, before you sell, re-evaluate your position. If you invested because you believe in the coin’s fundamentals, there are a few questions you can ask yourself:

Have any of the fundamentals changed?
Were there any announcements that would have affected the price?
Have you stopped believing in the long-term vision of the coin?

If your answer to all of these questions is “No”, then consider holding on. This strategy becomes much easier when you follow the golden rule of cryptocurrency trading: Don’t invest more money than you’re comfortable losing.

On the other side of this equation, seeing some solid gains may also tempt you to sell. Although taking profits is wise, you may want to avoid selling your entire stack. Depending on the situation, the coin could rise further. A popular trading strategy is to take out your initial investment while keeping your earnings invested in the coin after gaining a certain percentage. This decreases your downside risk while still exposing you to the upside potential.

Mistake #4: Being Uninformed

In a market that moves as rapidly as cryptocurrency does, you need to stay up-to-date with industry news. Without tuning in weekly, or even daily, the investment tides could shift without you even knowing.

Twitter, Reddit, and projects’ Telegram channels are also great resources you can use to stay informed. Oftentimes, teams share project updates and important announcements on these platforms before they hit mainstream media. Joining these communities also gives you the opportunity to be more involved with the projects while sometimes even impacting future development.

Good Luck Out There

Even with these tips, there’s bound to be mistakes that you make. Don’t let that discourage you – it happens to everyone. Part of the investing process is to learn from those mistakes and not make them again.

Continuous improvement is the name of the game. And, as long as you’ve got that going for you, you’ll be a trading whiz in no time.